[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:20:03 dist-test-slave-q9h9 krb5kdc[32573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:20:03 dist-test-slave-q9h9 krb5kdc[32573](info): set up 2 sockets
Aug 10 06:20:03 dist-test-slave-q9h9 krb5kdc[32573](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 06:20:05.079178 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.336s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:20:05 dist-test-slave-q9h9 krb5kdc[32573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:20:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:20:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:20:05.191099 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:38489
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:38489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:05.317272 32597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:05.317505 32597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:05.317544 32597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:05.321447 32597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:05.321496 32597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:05.321513 32597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:05.321534 32597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:05.321548 32597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:20:05.326885 32597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:38489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:38489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:05.328083 32597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:05.329252 32597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:05.342105 32608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:05.342242 32597 server_base.cc:1034] running on GCE node
W20260810 06:20:05.342878 32611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:05.343207 32607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:05.343940 32597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:05.345409 32597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:05.346568 32597 hybrid_clock.cc:648] HybridClock initialized: now 1786342805346523 us; error 56 us; skew 500 ppm
Aug 10 06:20:05 dist-test-slave-q9h9 krb5kdc[32573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342805, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:05.349954 32597 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:20:05.351054 32597 webserver.cc:460] Webserver started at http://127.31.204.254:39859/ using document root <none> and password file <none>
I20260810 06:20:05.351943 32597 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:05.352018 32597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:05.352425 32597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:05.355145 32597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "e635b6627d5941f5a0a5fe6acf6ee46b"
format_stamp: "Formatted at 2026-08-10 06:20:05 on dist-test-slave-q9h9"
server_key: "e87f8cfe4e6521002100afcfe85bb762"
server_key_iv: "2bea14e586bf284bb563f21d7e1bfc90"
server_key_version: "encryptionkey@0"
I20260810 06:20:05.356053 32597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "e635b6627d5941f5a0a5fe6acf6ee46b"
format_stamp: "Formatted at 2026-08-10 06:20:05 on dist-test-slave-q9h9"
server_key: "e87f8cfe4e6521002100afcfe85bb762"
server_key_iv: "2bea14e586bf284bb563f21d7e1bfc90"
server_key_version: "encryptionkey@0"
I20260810 06:20:05.361296 32597 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 06:20:05.364447 32618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:05.365919 32597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:20:05.366153 32597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "e635b6627d5941f5a0a5fe6acf6ee46b"
format_stamp: "Formatted at 2026-08-10 06:20:05 on dist-test-slave-q9h9"
server_key: "e87f8cfe4e6521002100afcfe85bb762"
server_key_iv: "2bea14e586bf284bb563f21d7e1bfc90"
server_key_version: "encryptionkey@0"
I20260810 06:20:05.366289 32597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:05.380264 32597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:05.383692 32597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:05.383908 32597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:05.399461 32597 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:38489
I20260810 06:20:05.399515 32712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:38489 every 8 connection(s)
I20260810 06:20:05.400759 32597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:20:05.405068 32713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:05.407963 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 32597
I20260810 06:20:05.408165 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:20:05.410175 32563 external_mini_cluster.cc:1428] Setting key c255a6d4644f0b2a0b2a85e5c2719d48
I20260810 06:20:05.416206 32713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b: Bootstrap starting.
I20260810 06:20:05.421804 32713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:05.423362 32713 log.cc:824] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:05.426229 32713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b: No bootstrap required, opened a new log
I20260810 06:20:05.432765 32713 raft_consensus.cc:348] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e635b6627d5941f5a0a5fe6acf6ee46b" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38489 } }
I20260810 06:20:05.433172 32713 raft_consensus.cc:374] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:05.433281 32713 raft_consensus.cc:729] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e635b6627d5941f5a0a5fe6acf6ee46b, State: Initialized, Role: FOLLOWER
I20260810 06:20:05.434048 32713 consensus_queue.cc:260] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e635b6627d5941f5a0a5fe6acf6ee46b" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38489 } }
I20260810 06:20:05.434280 32713 raft_consensus.cc:388] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:20:05.434428 32713 raft_consensus.cc:482] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:20:05.434592 32713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b [term 0 FOLLOWER]: Advancing to term 1
Aug 10 06:20:05 dist-test-slave-q9h9 krb5kdc[32573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:05.435855 32713 raft_consensus.cc:504] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e635b6627d5941f5a0a5fe6acf6ee46b" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38489 } }
I20260810 06:20:05.436496 32713 leader_election.cc:302] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e635b6627d5941f5a0a5fe6acf6ee46b; no voters:
I20260810 06:20:05.436883 32713 leader_election.cc:288] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:20:05.437044 32723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:05.437263 32723 raft_consensus.cc:686] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b [term 1 LEADER]: Becoming Leader. State: Replica: e635b6627d5941f5a0a5fe6acf6ee46b, State: Running, Role: LEADER
I20260810 06:20:05.437666 32723 consensus_queue.cc:237] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e635b6627d5941f5a0a5fe6acf6ee46b" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38489 } }
I20260810 06:20:05.438493 32713 sys_catalog.cc:564] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:05.440522 32724 sys_catalog.cc:455] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e635b6627d5941f5a0a5fe6acf6ee46b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e635b6627d5941f5a0a5fe6acf6ee46b" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38489 } } }
I20260810 06:20:05.440780 32724 sys_catalog.cc:458] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:05.441155 32727 sys_catalog.cc:455] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e635b6627d5941f5a0a5fe6acf6ee46b. Latest consensus state: current_term: 1 leader_uuid: "e635b6627d5941f5a0a5fe6acf6ee46b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e635b6627d5941f5a0a5fe6acf6ee46b" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38489 } } }
I20260810 06:20:05.441282 32727 sys_catalog.cc:458] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:05.442749 32736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:05.444051 32717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:05.415609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50592 (local address 127.31.204.254:38489)
0810 06:20:05.416213 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:05.416243 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:05.417341 (+ 1098us) server_negotiation.cc:407] Connection header received
0810 06:20:05.422877 (+ 5536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:05.422894 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:05.423247 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:05.423555 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:05.426844 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:05.428378 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:05.430487 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:05.431071 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:05.431219 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:05.436643 (+ 5424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:05.436720 (+ 77us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:05.436733 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:05.436763 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:05.439256 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:05.439944 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:05.439951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:05.439956 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:05.440110 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:05.440512 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:05.440515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:05.440517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:05.441256 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:05.441534 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:05.442928 (+ 1394us) server_negotiation.cc:299] Negotiation successful
0810 06:20:05.443233 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":199,"threads_started":1}
I20260810 06:20:05.447676 32736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:05.454700 32736 catalog_manager.cc:1347] Generated new cluster ID: 2b53269668e3434aa06285c885b54259
I20260810 06:20:05.454761 32736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:05.462086 32736 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:20:05.463330 32736 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:05.482213 32736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e635b6627d5941f5a0a5fe6acf6ee46b: Generated new TSK 0
I20260810 06:20:05.483167 32736 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
I20260810 06:20:05.512584 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:38489
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:05.642642 32755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:05.642854 32755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:05.642900 32755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:05.646718 32755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:05.646762 32755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:05.646842 32755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:20:05.651718 32755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:38489
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:05.652581 32755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:05.653481 32755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:05.662642 300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:05.662696 32767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:05.662667 302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:05.662716 32755 server_base.cc:1034] running on GCE node
I20260810 06:20:05.663285 32755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:05.663887 32755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:05.665047 32755 hybrid_clock.cc:648] HybridClock initialized: now 1786342805665033 us; error 24 us; skew 500 ppm
Aug 10 06:20:05 dist-test-slave-q9h9 krb5kdc[32573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342805, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:05.667608 32755 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:20:05.668561 32755 webserver.cc:460] Webserver started at http://127.31.204.193:43133/ using document root <none> and password file <none>
I20260810 06:20:05.669039 32755 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:05.669096 32755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:05.669286 32755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:05.671247 32755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "365c09fade5f42158964cc8f9fb69447"
format_stamp: "Formatted at 2026-08-10 06:20:05 on dist-test-slave-q9h9"
server_key: "daeb2c65fe345522c63c6d8b5615848b"
server_key_iv: "577a1bac4455dac47bf84d5f2345995c"
server_key_version: "encryptionkey@0"
I20260810 06:20:05.671835 32755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "365c09fade5f42158964cc8f9fb69447"
format_stamp: "Formatted at 2026-08-10 06:20:05 on dist-test-slave-q9h9"
server_key: "daeb2c65fe345522c63c6d8b5615848b"
server_key_iv: "577a1bac4455dac47bf84d5f2345995c"
server_key_version: "encryptionkey@0"
I20260810 06:20:05.675473 32755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:20:05.677570 316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:05.678655 32755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:05.678761 32755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "365c09fade5f42158964cc8f9fb69447"
format_stamp: "Formatted at 2026-08-10 06:20:05 on dist-test-slave-q9h9"
server_key: "daeb2c65fe345522c63c6d8b5615848b"
server_key_iv: "577a1bac4455dac47bf84d5f2345995c"
server_key_version: "encryptionkey@0"
I20260810 06:20:05.678849 32755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:05.692819 32755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:05.695156 32755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:05.695307 32755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:05.695847 32755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:05.696807 32755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:05.696861 32755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:05.696913 32755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:05.696940 32755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:05.711436 32755 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:32879
I20260810 06:20:05.711455 507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:32879 every 8 connection(s)
I20260810 06:20:05.712185 32755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:20:05.718190 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 32755
I20260810 06:20:05.718291 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:20:05.718555 32563 external_mini_cluster.cc:1428] Setting key f0c1064fd41e7f08ec1647a17c3faea1
Aug 10 06:20:05 dist-test-slave-q9h9 krb5kdc[32573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342805, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:05.725835 32717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:05.714071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:57893 (local address 127.31.204.254:38489)
0810 06:20:05.714205 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:05.714210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:05.715081 (+ 871us) server_negotiation.cc:407] Connection header received
0810 06:20:05.716538 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:05.716540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:05.716587 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:05.716661 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:05.717635 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:05.718214 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:05.719583 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:05.719925 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:05.720009 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:05.722599 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:05.722602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:05.722604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:05.722628 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:05.723915 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:05.724470 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:05.724472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:05.724474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:05.724542 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:05.724862 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:05.724865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:05.724867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:05.725069 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:05.725172 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:05.725565 (+ 393us) server_negotiation.cc:299] Negotiation successful
0810 06:20:05.725684 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:20:05.726455 511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:05.714434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:38489 (local address 127.31.204.193:57893)
0810 06:20:05.714940 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:05.714979 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:05.716358 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:05.716790 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:05.716796 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:05.717104 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:05.717527 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:05.717540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:05.718348 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:05.718351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:05.719473 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:05.719482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:05.720076 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:05.720080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:05.720262 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:05.720792 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:05.720816 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:05.722442 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:05.724025 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:05.724031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:05.724044 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:05.724383 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:05.724615 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:05.724618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:05.724620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:05.724761 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:05.725153 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:05.725157 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:05.725356 (+ 199us) client_negotiation.cc:769] Sending connection context
0810 06:20:05.725493 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 06:20:05.725703 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":180,"threads_started":1}
I20260810 06:20:05.727559 509 heartbeater.cc:347] Connected to a master server at 127.31.204.254:38489
I20260810 06:20:05.727753 509 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:05.728153 509 heartbeater.cc:511] Master 127.31.204.254:38489 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.204.194@KRBTEST.COM; defaulting to no policy
I20260810 06:20:05.729933 32645 ts_manager.cc:194] Registered new tserver with Master: 365c09fade5f42158964cc8f9fb69447 (127.31.204.193:32879)
I20260810 06:20:05.731400 32645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:57893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
I20260810 06:20:05.746382 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:38489
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:05.851765 515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:05.851958 515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:05.851996 515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:05.855861 515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:05.855906 515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:05.855960 515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:20:05.860857 515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:38489
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:05.861589 515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:05.862468 515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:05.870862 524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:05.870872 527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:05.870898 535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:05.871043 515 server_base.cc:1034] running on GCE node
I20260810 06:20:05.871461 515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:05.871994 515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:05.873142 515 hybrid_clock.cc:648] HybridClock initialized: now 1786342805873127 us; error 29 us; skew 500 ppm
Aug 10 06:20:05 dist-test-slave-q9h9 krb5kdc[32573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342805, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:05.875691 515 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:20:05.876622 515 webserver.cc:460] Webserver started at http://127.31.204.194:37961/ using document root <none> and password file <none>
I20260810 06:20:05.877074 515 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:05.877130 515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:05.877316 515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:05.879110 515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "437a1e7321ee4574baf5c94cb95c9db1"
format_stamp: "Formatted at 2026-08-10 06:20:05 on dist-test-slave-q9h9"
server_key: "8978ee3dccc7fb8a600065308577e4dc"
server_key_iv: "7d7d5191873337b445dcb89ace1ad5ac"
server_key_version: "encryptionkey@0"
I20260810 06:20:05.879614 515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "437a1e7321ee4574baf5c94cb95c9db1"
format_stamp: "Formatted at 2026-08-10 06:20:05 on dist-test-slave-q9h9"
server_key: "8978ee3dccc7fb8a600065308577e4dc"
server_key_iv: "7d7d5191873337b445dcb89ace1ad5ac"
server_key_version: "encryptionkey@0"
I20260810 06:20:05.883332 515 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:20:05.885468 548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:05.886554 515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:05.886662 515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "437a1e7321ee4574baf5c94cb95c9db1"
format_stamp: "Formatted at 2026-08-10 06:20:05 on dist-test-slave-q9h9"
server_key: "8978ee3dccc7fb8a600065308577e4dc"
server_key_iv: "7d7d5191873337b445dcb89ace1ad5ac"
server_key_version: "encryptionkey@0"
I20260810 06:20:05.886746 515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:05.898674 515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:05.901340 515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:05.901494 515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:05.902045 515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:05.902786 515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:05.902832 515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:05.902880 515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:05.902906 515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:05.917936 515 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:44839
I20260810 06:20:05.917960 722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:44839 every 8 connection(s)
I20260810 06:20:05.918706 515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:20:05.923303 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 515
I20260810 06:20:05.923421 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:20:05.923702 32563 external_mini_cluster.cc:1428] Setting key a352c417e6edd1a04a2a4f1aaf5dcef6
Aug 10 06:20:05 dist-test-slave-q9h9 krb5kdc[32573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342805, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:05.931610 32717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:05.920370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:46759 (local address 127.31.204.254:38489)
0810 06:20:05.920533 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:05.920538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:05.921329 (+ 791us) server_negotiation.cc:407] Connection header received
0810 06:20:05.922531 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:05.922534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:05.922583 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:05.922655 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:05.923891 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:05.924414 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:05.925483 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:05.925713 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:05.925801 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:05.928232 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:05.928236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:05.928239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:05.928269 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:05.929603 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:05.930190 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:05.930192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:05.930193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:05.930267 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:05.930600 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:05.930602 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:05.930604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:05.930757 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:05.930825 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:05.931393 (+ 568us) server_negotiation.cc:299] Negotiation successful
0810 06:20:05.931497 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 06:20:05.932533 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:05.920665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:38489 (local address 127.31.204.194:46759)
0810 06:20:05.921178 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:05.921225 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:05.922340 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:05.922784 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:05.922790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:05.923145 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:05.923735 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:05.923756 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:05.924539 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:05.924542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:05.925367 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:05.925377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:05.925847 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:05.925851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:05.926000 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:05.926509 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:05.926524 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:05.928047 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:05.929715 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:05.929720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:05.929733 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:05.930093 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:05.930347 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:05.930350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:05.930352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:05.930517 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:05.930847 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:05.930852 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:05.931109 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 06:20:05.931283 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 06:20:05.931614 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":200,"threads_started":1}
I20260810 06:20:05.933571 723 heartbeater.cc:347] Connected to a master server at 127.31.204.254:38489
I20260810 06:20:05.933811 723 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:05.934343 723 heartbeater.cc:511] Master 127.31.204.254:38489 requested a full tablet report, sending...
I20260810 06:20:05.935385 32645 ts_manager.cc:194] Registered new tserver with Master: 437a1e7321ee4574baf5c94cb95c9db1 (127.31.204.194:44839)
I20260810 06:20:05.935920 32645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:46759
WARNING: no policy specified for kudu/127.31.204.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
I20260810 06:20:05.950841 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:38489
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:06.056140 733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:06.056342 733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:06.056387 733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:06.060292 733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:06.060340 733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:06.060410 733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:20:06.065326 733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:38489
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:06.066176 733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:06.067032 733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:06.075599 747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:06.075628 744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:06.075745 733 server_base.cc:1034] running on GCE node
W20260810 06:20:06.075628 743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:06.076261 733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:06.076789 733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:06.077955 733 hybrid_clock.cc:648] HybridClock initialized: now 1786342806077940 us; error 26 us; skew 500 ppm
Aug 10 06:20:06 dist-test-slave-q9h9 krb5kdc[32573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342806, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:06.080469 733 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:20:06.081288 733 webserver.cc:460] Webserver started at http://127.31.204.195:46591/ using document root <none> and password file <none>
I20260810 06:20:06.081753 733 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:06.081832 733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:06.082093 733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:06.083927 733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "9c2f49b7e3ed4c9dbff34f2527d42025"
format_stamp: "Formatted at 2026-08-10 06:20:06 on dist-test-slave-q9h9"
server_key: "e241fb2f58b0f6339a13ced8381c9649"
server_key_iv: "5d9eb04eb685a9fd5a04f31f9694af2f"
server_key_version: "encryptionkey@0"
I20260810 06:20:06.084558 733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "9c2f49b7e3ed4c9dbff34f2527d42025"
format_stamp: "Formatted at 2026-08-10 06:20:06 on dist-test-slave-q9h9"
server_key: "e241fb2f58b0f6339a13ced8381c9649"
server_key_iv: "5d9eb04eb685a9fd5a04f31f9694af2f"
server_key_version: "encryptionkey@0"
I20260810 06:20:06.088107 733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:20:06.090376 757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:06.091486 733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:06.091578 733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "9c2f49b7e3ed4c9dbff34f2527d42025"
format_stamp: "Formatted at 2026-08-10 06:20:06 on dist-test-slave-q9h9"
server_key: "e241fb2f58b0f6339a13ced8381c9649"
server_key_iv: "5d9eb04eb685a9fd5a04f31f9694af2f"
server_key_version: "encryptionkey@0"
I20260810 06:20:06.091666 733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:06.107553 733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:06.109961 733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:06.110127 733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:06.110602 733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:06.111305 733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:06.111351 733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:06.111387 733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:06.111406 733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:06.127187 733 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:43953
I20260810 06:20:06.127206 932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:43953 every 8 connection(s)
I20260810 06:20:06.128034 733 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:20:06.135681 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 733
I20260810 06:20:06.135803 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:20:06.136049 32563 external_mini_cluster.cc:1428] Setting key c86bd105729adc19b039e4f21236bc63
Aug 10 06:20:06 dist-test-slave-q9h9 krb5kdc[32573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342806, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:06.140749 32717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:06.129795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:57269 (local address 127.31.204.254:38489)
0810 06:20:06.129917 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:06.129921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:06.130709 (+ 788us) server_negotiation.cc:407] Connection header received
0810 06:20:06.131994 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:06.131998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:06.132055 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:06.132139 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:06.133217 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:06.133740 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:06.134904 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:06.135132 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:06.135222 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:06.137619 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:06.137622 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:06.137624 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:06.137647 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:06.139055 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:06.139500 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:06.139502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:06.139503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:06.139569 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:06.139836 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:06.139839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:06.139840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:06.140024 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:06.140124 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:06.140546 (+ 422us) server_negotiation.cc:299] Negotiation successful
0810 06:20:06.140639 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:20:06.141506 935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:06.130134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:38489 (local address 127.31.204.195:57269)
0810 06:20:06.130563 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:06.130604 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:06.131799 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:06.132299 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:06.132306 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:06.132646 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:06.133078 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:06.133091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:06.133857 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:06.133860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:06.134773 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:06.134784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:06.135271 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:06.135275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:06.135426 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:06.135965 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:06.135980 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:06.137492 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:06.139166 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:06.139171 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:06.139181 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:06.139423 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:06.139634 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:06.139637 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:06.139638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:06.139776 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:06.140139 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:06.140145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:06.140357 (+ 212us) client_negotiation.cc:769] Sending connection context
0810 06:20:06.140502 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 06:20:06.140736 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":120,"threads_started":1}
I20260810 06:20:06.142508 933 heartbeater.cc:347] Connected to a master server at 127.31.204.254:38489
I20260810 06:20:06.142710 933 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:06.143143 933 heartbeater.cc:511] Master 127.31.204.254:38489 requested a full tablet report, sending...
I20260810 06:20:06.144078 32645 ts_manager.cc:194] Registered new tserver with Master: 9c2f49b7e3ed4c9dbff34f2527d42025 (127.31.204.195:43953)
I20260810 06:20:06.144596 32645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:57269
I20260810 06:20:06.150249 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 06:20:06 dist-test-slave-q9h9 krb5kdc[32573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342806, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 06:20:06 dist-test-slave-q9h9 krb5kdc[32573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342806, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.204.193@KRBTEST.COM
I20260810 06:20:06.178103 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:06.169430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45294 (local address 127.31.204.193:32879)
0810 06:20:06.169774 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:06.169784 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:06.169804 (+ 20us) server_negotiation.cc:407] Connection header received
0810 06:20:06.169846 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:06.169850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:06.170065 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:06.170181 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:06.170858 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:06.171630 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:06.172489 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:06.172835 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:06.172935 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:06.175079 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:06.175088 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:06.175098 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:06.175120 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:06.176379 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:06.176757 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:06.176763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:06.176765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:06.176842 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:06.177078 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:06.177082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:06.177084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:06.177343 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:06.177543 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:06.177721 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 06:20:06.177863 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":103,"threads_started":1}
W20260810 06:20:06.178988 359 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45294
Aug 10 06:20:06 dist-test-slave-q9h9 krb5kdc[32573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 06:20:06 dist-test-slave-q9h9 krb5kdc[32573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.193@KRBTEST.COM
I20260810 06:20:06.203343 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:06.196574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.31.204.193:32879)
0810 06:20:06.196689 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:06.196693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:06.196831 (+ 138us) server_negotiation.cc:407] Connection header received
0810 06:20:06.196917 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:06.196920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:06.196954 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:06.197013 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:06.197608 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:06.198125 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:06.198844 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:06.199052 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:06.199116 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:06.201080 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:06.201083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:06.201085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:06.201106 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:06.202154 (+ 1048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:06.202491 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:06.202493 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:06.202494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:06.202556 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:06.202722 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:06.202723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:06.202724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:06.202905 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:06.202996 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:06.203150 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 06:20:06.203237 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:20:06.204255 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 32755
I20260810 06:20:06.216660 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 515
I20260810 06:20:06.228971 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 733
I20260810 06:20:06.241976 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 32597
2026-08-10T06:20:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2538 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:20:06 dist-test-slave-q9h9 krb5kdc[952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:20:06 dist-test-slave-q9h9 krb5kdc[952](info): set up 2 sockets
Aug 10 06:20:06 dist-test-slave-q9h9 krb5kdc[952](info): commencing operation
krb5kdc: starting...
W20260810 06:20:07.411918 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:20:07 dist-test-slave-q9h9 krb5kdc[952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:20:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:20:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:20:07.522074 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:35619
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:35619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:07.632819 970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:07.633035 970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:07.633072 970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:07.636960 970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:07.637007 970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:07.637025 970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:07.637041 970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:07.637068 970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:20:07.641991 970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:35619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:35619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:07.642880 970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:07.643795 970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:20:07.651479 970 server_base.cc:1034] running on GCE node
W20260810 06:20:07.651408 977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:07.651441 981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:07.651427 978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:07.652146 970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:07.653101 970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:07.654247 970 hybrid_clock.cc:648] HybridClock initialized: now 1786342807654235 us; error 28 us; skew 500 ppm
Aug 10 06:20:07 dist-test-slave-q9h9 krb5kdc[952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342807, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:07.659838 970 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:20:07.660615 970 webserver.cc:460] Webserver started at http://127.31.204.254:39311/ using document root <none> and password file <none>
I20260810 06:20:07.661101 970 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:07.661165 970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:07.661335 970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:07.663216 970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "173e7c6f9f524403be2f68f0d1e9e09c"
format_stamp: "Formatted at 2026-08-10 06:20:07 on dist-test-slave-q9h9"
server_key: "ea30b0d5cc6b49d303d594ff0224fe01"
server_key_iv: "f152147d2a409175231c0d50ff314911"
server_key_version: "encryptionkey@0"
I20260810 06:20:07.663720 970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "173e7c6f9f524403be2f68f0d1e9e09c"
format_stamp: "Formatted at 2026-08-10 06:20:07 on dist-test-slave-q9h9"
server_key: "ea30b0d5cc6b49d303d594ff0224fe01"
server_key_iv: "f152147d2a409175231c0d50ff314911"
server_key_version: "encryptionkey@0"
I20260810 06:20:07.667227 970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:20:07.669404 996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:07.670554 970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:07.670667 970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "173e7c6f9f524403be2f68f0d1e9e09c"
format_stamp: "Formatted at 2026-08-10 06:20:07 on dist-test-slave-q9h9"
server_key: "ea30b0d5cc6b49d303d594ff0224fe01"
server_key_iv: "f152147d2a409175231c0d50ff314911"
server_key_version: "encryptionkey@0"
I20260810 06:20:07.670750 970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:07.683918 970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:07.686632 970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:07.686795 970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:07.693143 970 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:35619
I20260810 06:20:07.693154 1069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:35619 every 8 connection(s)
I20260810 06:20:07.693974 970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:20:07.696486 1070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:07.696918 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 970
I20260810 06:20:07.697027 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:20:07.697280 32563 external_mini_cluster.cc:1428] Setting key c01a9affe64163f929ffbed5280ed42b
I20260810 06:20:07.702809 1070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c: Bootstrap starting.
I20260810 06:20:07.705403 1070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c: Neither blocks nor log segments found. Creating new log.
Aug 10 06:20:07 dist-test-slave-q9h9 krb5kdc[952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:07.706243 1070 log.cc:824] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:07.707971 1070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c: No bootstrap required, opened a new log
I20260810 06:20:07.710764 1075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:07.698512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48738 (local address 127.31.204.254:35619)
0810 06:20:07.698875 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:07.698889 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:07.698921 (+ 32us) server_negotiation.cc:407] Connection header received
0810 06:20:07.699539 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:07.699554 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:07.699861 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:07.700105 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:07.700931 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:07.701830 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:07.703020 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:07.703731 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:07.703879 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:07.706716 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:07.706722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:07.706737 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:07.706764 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:07.708308 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:07.708744 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:07.708748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:07.708752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:07.708849 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:07.709105 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:07.709107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:07.709108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:07.709441 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:07.709569 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:07.709845 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 06:20:07.710113 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":93,"threads_started":1}
I20260810 06:20:07.711833 1070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173e7c6f9f524403be2f68f0d1e9e09c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35619 } }
I20260810 06:20:07.712023 1070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:07.712077 1070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 173e7c6f9f524403be2f68f0d1e9e09c, State: Initialized, Role: FOLLOWER
I20260810 06:20:07.712430 1070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173e7c6f9f524403be2f68f0d1e9e09c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35619 } }
I20260810 06:20:07.712548 1070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:20:07.712610 1070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:20:07.712698 1070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:07.713512 1070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173e7c6f9f524403be2f68f0d1e9e09c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35619 } }
I20260810 06:20:07.713801 1070 leader_election.cc:302] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 173e7c6f9f524403be2f68f0d1e9e09c; no voters:
I20260810 06:20:07.714073 1070 leader_election.cc:288] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:20:07.714154 1079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:07.714361 1079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c [term 1 LEADER]: Becoming Leader. State: Replica: 173e7c6f9f524403be2f68f0d1e9e09c, State: Running, Role: LEADER
I20260810 06:20:07.714630 1079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173e7c6f9f524403be2f68f0d1e9e09c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35619 } }
I20260810 06:20:07.715128 1070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:07.716152 1080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "173e7c6f9f524403be2f68f0d1e9e09c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173e7c6f9f524403be2f68f0d1e9e09c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35619 } } }
I20260810 06:20:07.716281 1080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:07.716657 1095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:07.716567 1082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 173e7c6f9f524403be2f68f0d1e9e09c. Latest consensus state: current_term: 1 leader_uuid: "173e7c6f9f524403be2f68f0d1e9e09c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173e7c6f9f524403be2f68f0d1e9e09c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35619 } } }
I20260810 06:20:07.717110 1082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:07.719272 1095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:07.723915 1095 catalog_manager.cc:1347] Generated new cluster ID: 8b01acffbc154bb19e8a812161d5f216
I20260810 06:20:07.723982 1095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:07.737146 1095 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:20:07.738243 1095 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:07.744084 1095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 173e7c6f9f524403be2f68f0d1e9e09c: Generated new TSK 0
I20260810 06:20:07.744699 1095 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
I20260810 06:20:07.777859 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35619
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:07.882436 1110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:07.882635 1110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:07.882675 1110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:07.886493 1110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:07.886538 1110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:07.886591 1110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:20:07.891417 1110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35619
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:07.892268 1110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:07.893179 1110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:20:07.902120 1110 server_base.cc:1034] running on GCE node
W20260810 06:20:07.903735 1123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:07.904139 1121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:07.904366 1120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:07.904739 1110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:07.905316 1110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:07.906473 1110 hybrid_clock.cc:648] HybridClock initialized: now 1786342807906458 us; error 26 us; skew 500 ppm
Aug 10 06:20:07 dist-test-slave-q9h9 krb5kdc[952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342807, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:07.908895 1110 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:20:07.909709 1110 webserver.cc:460] Webserver started at http://127.31.204.193:46519/ using document root <none> and password file <none>
I20260810 06:20:07.910240 1110 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:07.910308 1110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:07.910495 1110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:07.912317 1110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "404f1cff07374d0f8e9a082227bff1c6"
format_stamp: "Formatted at 2026-08-10 06:20:07 on dist-test-slave-q9h9"
server_key: "e0291ba1d4258d6280a22235e9f8f931"
server_key_iv: "3407bf1bf944adb3b7289deb15a8b96d"
server_key_version: "encryptionkey@0"
I20260810 06:20:07.912829 1110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "404f1cff07374d0f8e9a082227bff1c6"
format_stamp: "Formatted at 2026-08-10 06:20:07 on dist-test-slave-q9h9"
server_key: "e0291ba1d4258d6280a22235e9f8f931"
server_key_iv: "3407bf1bf944adb3b7289deb15a8b96d"
server_key_version: "encryptionkey@0"
I20260810 06:20:07.916239 1110 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260810 06:20:07.918438 1130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:07.919477 1110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:07.919579 1110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "404f1cff07374d0f8e9a082227bff1c6"
format_stamp: "Formatted at 2026-08-10 06:20:07 on dist-test-slave-q9h9"
server_key: "e0291ba1d4258d6280a22235e9f8f931"
server_key_iv: "3407bf1bf944adb3b7289deb15a8b96d"
server_key_version: "encryptionkey@0"
I20260810 06:20:07.919660 1110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:07.932741 1110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:07.935142 1110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:07.935293 1110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:07.935784 1110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:07.936488 1110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:07.936533 1110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:07.936581 1110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:07.936611 1110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:07.944686 1110 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:43397
I20260810 06:20:07.944715 1328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:43397 every 8 connection(s)
I20260810 06:20:07.945418 1110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:20:07.953037 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 1110
I20260810 06:20:07.953119 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:20:07.953341 32563 external_mini_cluster.cc:1428] Setting key ca03318bfe0fa748aa88081fc3d2d31b
Aug 10 06:20:07 dist-test-slave-q9h9 krb5kdc[952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342807, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:07.957427 1075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:07.946956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:53249 (local address 127.31.204.254:35619)
0810 06:20:07.947103 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:07.947108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:07.947850 (+ 742us) server_negotiation.cc:407] Connection header received
0810 06:20:07.949006 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:07.949009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:07.949054 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:07.949120 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:07.950226 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:07.950746 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:07.951758 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:07.951976 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:07.952039 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:07.954261 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:07.954264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:07.954266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:07.954308 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:07.955626 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:07.956074 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:07.956077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:07.956079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:07.956163 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:07.956460 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:07.956463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:07.956464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:07.956632 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:07.956714 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:07.957152 (+ 438us) server_negotiation.cc:299] Negotiation successful
0810 06:20:07.957271 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 06:20:07.958138 1334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:07.947289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:35619 (local address 127.31.204.193:53249)
0810 06:20:07.947719 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:07.947758 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:07.948836 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:07.949275 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:07.949280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:07.949578 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:07.950110 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:07.950125 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:07.950838 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:07.950841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:07.951666 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:07.951675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:07.952072 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:07.952074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:07.952221 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:07.952682 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:07.952697 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:07.954101 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:07.955733 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:07.955739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:07.955751 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:07.955982 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:07.956243 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:07.956246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:07.956248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:07.956375 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:07.956728 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:07.956734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:07.956938 (+ 204us) client_negotiation.cc:769] Sending connection context
0810 06:20:07.957086 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 06:20:07.957360 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":103,"threads_started":1}
I20260810 06:20:07.959267 1331 heartbeater.cc:347] Connected to a master server at 127.31.204.254:35619
I20260810 06:20:07.959458 1331 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:07.959851 1331 heartbeater.cc:511] Master 127.31.204.254:35619 requested a full tablet report, sending...
I20260810 06:20:07.961328 1024 ts_manager.cc:194] Registered new tserver with Master: 404f1cff07374d0f8e9a082227bff1c6 (127.31.204.193:43397)
I20260810 06:20:07.962888 1024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:53249
WARNING: no policy specified for kudu/127.31.204.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
I20260810 06:20:07.977285 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35619
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:08.079638 1339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:08.079839 1339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:08.079878 1339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:08.083709 1339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:08.083756 1339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:08.083810 1339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:20:08.088593 1339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35619
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:08.089375 1339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:08.090241 1339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:08.098763 1345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:08.098755 1349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:08.098757 1347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:08.098960 1339 server_base.cc:1034] running on GCE node
I20260810 06:20:08.099342 1339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:08.099901 1339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:08.101063 1339 hybrid_clock.cc:648] HybridClock initialized: now 1786342808101047 us; error 29 us; skew 500 ppm
Aug 10 06:20:08 dist-test-slave-q9h9 krb5kdc[952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342808, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:08.103484 1339 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:20:08.104290 1339 webserver.cc:460] Webserver started at http://127.31.204.194:32935/ using document root <none> and password file <none>
I20260810 06:20:08.104756 1339 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:08.104818 1339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:08.104997 1339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:08.106832 1339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "93141253bf8b4bf1adb059f3e9db29fb"
format_stamp: "Formatted at 2026-08-10 06:20:08 on dist-test-slave-q9h9"
server_key: "11a002c8d21e969a025c1273ff53debc"
server_key_iv: "e8c450c795f1510f6c2772522a530eea"
server_key_version: "encryptionkey@0"
I20260810 06:20:08.107369 1339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "93141253bf8b4bf1adb059f3e9db29fb"
format_stamp: "Formatted at 2026-08-10 06:20:08 on dist-test-slave-q9h9"
server_key: "11a002c8d21e969a025c1273ff53debc"
server_key_iv: "e8c450c795f1510f6c2772522a530eea"
server_key_version: "encryptionkey@0"
I20260810 06:20:08.110648 1339 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:20:08.112726 1358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:08.113768 1339 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:08.113869 1339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "93141253bf8b4bf1adb059f3e9db29fb"
format_stamp: "Formatted at 2026-08-10 06:20:08 on dist-test-slave-q9h9"
server_key: "11a002c8d21e969a025c1273ff53debc"
server_key_iv: "e8c450c795f1510f6c2772522a530eea"
server_key_version: "encryptionkey@0"
I20260810 06:20:08.113951 1339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:08.135521 1339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:08.137877 1339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:08.138053 1339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:08.138566 1339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:08.139302 1339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:08.139348 1339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:08.139397 1339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:08.139426 1339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:08.147541 1339 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:46443
I20260810 06:20:08.147576 1526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:46443 every 8 connection(s)
I20260810 06:20:08.148324 1339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:20:08.152212 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 1339
I20260810 06:20:08.152330 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:20:08.152567 32563 external_mini_cluster.cc:1428] Setting key 3b8a28e2f834bcb028763859d579f496
Aug 10 06:20:08 dist-test-slave-q9h9 krb5kdc[952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342808, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:08.161737 1075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:08.149910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:43133 (local address 127.31.204.254:35619)
0810 06:20:08.150041 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:08.150044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:08.150805 (+ 761us) server_negotiation.cc:407] Connection header received
0810 06:20:08.152005 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:08.152009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:08.152066 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:08.152168 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:08.153387 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.154122 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.155599 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.155812 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.155890 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:08.158202 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:08.158207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:08.158209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:08.158242 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:08.159838 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:08.160308 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:08.160312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:08.160313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:08.160407 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:08.160686 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:08.160689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:08.160691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:08.160918 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:08.161018 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:08.161437 (+ 419us) server_negotiation.cc:299] Negotiation successful
0810 06:20:08.161557 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":43}
I20260810 06:20:08.162352 1531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:08.150249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:35619 (local address 127.31.204.194:43133)
0810 06:20:08.150659 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:08.150699 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:08.151819 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:08.152294 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:08.152302 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:08.152662 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:08.153257 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:08.153272 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.154249 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.154253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:08.155488 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:08.155501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.155941 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.155945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:08.156134 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:08.156641 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:08.156657 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:08.157993 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:08.159948 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:08.159952 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:08.159962 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:08.160215 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:08.160489 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:08.160490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:08.160492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:08.160604 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:08.161005 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:08.161009 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:08.161217 (+ 208us) client_negotiation.cc:769] Sending connection context
0810 06:20:08.161356 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 06:20:08.161571 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.31.204.195@KRBTEST.COM; defaulting to no policy
I20260810 06:20:08.163321 1529 heartbeater.cc:347] Connected to a master server at 127.31.204.254:35619
I20260810 06:20:08.163508 1529 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:08.163934 1529 heartbeater.cc:511] Master 127.31.204.254:35619 requested a full tablet report, sending...
I20260810 06:20:08.164851 1024 ts_manager.cc:194] Registered new tserver with Master: 93141253bf8b4bf1adb059f3e9db29fb (127.31.204.194:46443)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.195@KRBTEST.COM" created.
I20260810 06:20:08.165447 1024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:43133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
I20260810 06:20:08.178457 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35619
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:08.282526 1535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:08.282734 1535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:08.282780 1535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:08.286569 1535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:08.286614 1535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:08.286667 1535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:20:08.291455 1535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35619
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:08.292255 1535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:08.293100 1535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:08.301625 1541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:08.301713 1542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:08.301654 1546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:08.301774 1535 server_base.cc:1034] running on GCE node
I20260810 06:20:08.302189 1535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:08.302736 1535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:08.303886 1535 hybrid_clock.cc:648] HybridClock initialized: now 1786342808303870 us; error 26 us; skew 500 ppm
Aug 10 06:20:08 dist-test-slave-q9h9 krb5kdc[952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342808, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:08.306392 1535 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:20:08.307209 1535 webserver.cc:460] Webserver started at http://127.31.204.195:34463/ using document root <none> and password file <none>
I20260810 06:20:08.307693 1535 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:08.307757 1535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:08.307929 1535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:08.309814 1535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "55a32e9e7f504d56a31355d49e1cea5e"
format_stamp: "Formatted at 2026-08-10 06:20:08 on dist-test-slave-q9h9"
server_key: "f3a31ad50051432fe575ef3a2bf3083e"
server_key_iv: "7079efeda5c7007a826b78311a5bc320"
server_key_version: "encryptionkey@0"
I20260810 06:20:08.310377 1535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "55a32e9e7f504d56a31355d49e1cea5e"
format_stamp: "Formatted at 2026-08-10 06:20:08 on dist-test-slave-q9h9"
server_key: "f3a31ad50051432fe575ef3a2bf3083e"
server_key_iv: "7079efeda5c7007a826b78311a5bc320"
server_key_version: "encryptionkey@0"
I20260810 06:20:08.313634 1535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:20:08.315922 1556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:08.317056 1535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:08.317157 1535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "55a32e9e7f504d56a31355d49e1cea5e"
format_stamp: "Formatted at 2026-08-10 06:20:08 on dist-test-slave-q9h9"
server_key: "f3a31ad50051432fe575ef3a2bf3083e"
server_key_iv: "7079efeda5c7007a826b78311a5bc320"
server_key_version: "encryptionkey@0"
I20260810 06:20:08.317246 1535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:08.337177 1535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:08.339920 1535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:08.340072 1535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:08.340574 1535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:08.341359 1535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:08.341413 1535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:08.341461 1535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:08.341485 1535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:08.350111 1535 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:41381
I20260810 06:20:08.350137 1730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:41381 every 8 connection(s)
I20260810 06:20:08.350860 1535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:20:08.353345 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 1535
I20260810 06:20:08.353456 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:20:08.353724 32563 external_mini_cluster.cc:1428] Setting key d98930ff2a7b6905cf5fc51001d92214
Aug 10 06:20:08 dist-test-slave-q9h9 krb5kdc[952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342808, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:08.362798 1075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:08.352573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:52867 (local address 127.31.204.254:35619)
0810 06:20:08.352716 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:08.352719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:08.353500 (+ 781us) server_negotiation.cc:407] Connection header received
0810 06:20:08.354984 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:08.354986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:08.355037 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:08.355110 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:08.356181 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.356688 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.357716 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.357924 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.357987 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:08.360057 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:08.360060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:08.360062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:08.360086 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:08.361221 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:08.361637 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:08.361639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:08.361640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:08.361705 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:08.361939 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:08.361941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:08.361942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:08.362137 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:08.362208 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:08.362596 (+ 388us) server_negotiation.cc:299] Negotiation successful
0810 06:20:08.362680 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 06:20:08.363484 1737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:08.352893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:35619 (local address 127.31.204.195:52867)
0810 06:20:08.353361 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:08.353403 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:08.354771 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:08.355253 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:08.355260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:08.355620 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:08.356058 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:08.356071 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.356774 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.356777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:08.357620 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:08.357629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.358045 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.358050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:08.358208 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:08.358624 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:08.358639 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:08.359909 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:08.361322 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:08.361326 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:08.361336 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:08.361563 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:08.361762 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:08.361764 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:08.361766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:08.361877 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:08.362220 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:08.362224 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:08.362419 (+ 195us) client_negotiation.cc:769] Sending connection context
0810 06:20:08.362557 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 06:20:08.362768 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":109,"threads_started":1}
I20260810 06:20:08.364449 1732 heartbeater.cc:347] Connected to a master server at 127.31.204.254:35619
I20260810 06:20:08.364640 1732 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:08.365046 1732 heartbeater.cc:511] Master 127.31.204.254:35619 requested a full tablet report, sending...
I20260810 06:20:08.365908 1024 ts_manager.cc:194] Registered new tserver with Master: 55a32e9e7f504d56a31355d49e1cea5e (127.31.204.195:41381)
I20260810 06:20:08.366569 1024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:52867
I20260810 06:20:08.367271 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:20:08.377225 1075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:08.371068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46078 (local address 127.31.204.254:35619)
0810 06:20:08.371172 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:08.371175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:08.371337 (+ 162us) server_negotiation.cc:407] Connection header received
0810 06:20:08.371452 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:08.371455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:08.371489 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:08.371543 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:08.372155 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.372613 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.373396 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.373592 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.373671 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:08.374821 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:08.374825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:08.374827 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:08.374847 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:08.375826 (+ 979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:08.376190 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:08.376193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:08.376195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:08.376262 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:08.376492 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:08.376495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:08.376497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:08.376676 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:08.376807 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:08.377005 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 06:20:08.377098 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 06:20:08.382081 1022 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:20:08.385018 1022 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:20:08.400094 1758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:08.394074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:46443 (local address 127.0.0.1:58168)
0810 06:20:08.394853 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:08.394866 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:08.394981 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:08.396005 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:08.396007 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:08.396021 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:08.396120 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:08.396126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.397212 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.397234 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:08.398649 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:08.398660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.399715 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.399719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:08.399819 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:08.399841 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:20:08.399887 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 06:20:08.399949 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":79,"thread_start_us":136,"threads_started":1}
I20260810 06:20:08.400182 1761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:08.394882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55302 (local address 127.31.204.193:43397)
0810 06:20:08.395281 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:08.395295 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:08.395563 (+ 268us) server_negotiation.cc:407] Connection header received
0810 06:20:08.395649 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:08.395654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:08.395833 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:08.395964 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:08.396214 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.397270 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.398593 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.399363 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.399484 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:08.399664 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:08.399839 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 06:20:08.399962 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":103,"threads_started":1}
I20260810 06:20:08.400588 1762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:08.395184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58168 (local address 127.31.204.194:46443)
0810 06:20:08.395581 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:08.395591 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:08.395614 (+ 23us) server_negotiation.cc:407] Connection header received
0810 06:20:08.395683 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:08.395688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:08.395870 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:08.396259 (+ 389us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:08.396386 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.397058 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.398777 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.399581 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.400178 (+ 597us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:08.400316 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:08.400363 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 06:20:08.400446 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":191,"threads_started":1}
I20260810 06:20:08.400081 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:08.394689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:43397 (local address 127.0.0.1:55302)
0810 06:20:08.395385 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:08.395396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:08.395479 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:08.395967 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:08.395969 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:08.395987 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:08.396106 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:08.396113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.397407 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.397410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:08.398484 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:08.398494 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.399491 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.399494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:08.399616 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:08.399686 (+ 70us) client_negotiation.cc:769] Sending connection context
0810 06:20:08.399796 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 06:20:08.399907 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":53,"thread_start_us":187,"threads_started":1}
I20260810 06:20:08.401089 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:08.393985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:41381 (local address 127.0.0.1:59098)
0810 06:20:08.394509 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:08.394552 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:08.395193 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:08.395677 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:08.395683 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:08.395706 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:08.395825 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:08.395831 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.397162 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.397165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:08.398777 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:08.398785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.400724 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.400728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:08.400829 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:08.400852 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 06:20:08.400908 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 06:20:08.400972 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":181,"thread_start_us":74,"threads_started":1}
I20260810 06:20:08.401618 1757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:08.394083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59098 (local address 127.31.204.195:41381)
0810 06:20:08.394370 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:08.394381 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:08.394645 (+ 264us) server_negotiation.cc:407] Connection header received
0810 06:20:08.395318 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:08.395324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:08.395514 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:08.395662 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:08.395936 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.397018 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.398879 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.399467 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.401261 (+ 1794us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:08.401368 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:08.401412 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 06:20:08.401487 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":119,"threads_started":1}
I20260810 06:20:08.403487 1208 tablet_service.cc:1467] Processing CreateTablet for tablet 523f747d5a8348a89c2799324ce905ee (DEFAULT_TABLE table=test-table [id=4f066fa17d3c436e944bd2900b63ed54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:08.403663 1424 tablet_service.cc:1467] Processing CreateTablet for tablet 523f747d5a8348a89c2799324ce905ee (DEFAULT_TABLE table=test-table [id=4f066fa17d3c436e944bd2900b63ed54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:08.403924 1630 tablet_service.cc:1467] Processing CreateTablet for tablet 523f747d5a8348a89c2799324ce905ee (DEFAULT_TABLE table=test-table [id=4f066fa17d3c436e944bd2900b63ed54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:08.404532 1208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 523f747d5a8348a89c2799324ce905ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:08.404646 1424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 523f747d5a8348a89c2799324ce905ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:08.404767 1630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 523f747d5a8348a89c2799324ce905ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:08.410815 1763 tablet_bootstrap.cc:492] T 523f747d5a8348a89c2799324ce905ee P 404f1cff07374d0f8e9a082227bff1c6: Bootstrap starting.
I20260810 06:20:08.411895 1769 tablet_bootstrap.cc:492] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb: Bootstrap starting.
I20260810 06:20:08.412436 1768 tablet_bootstrap.cc:492] T 523f747d5a8348a89c2799324ce905ee P 55a32e9e7f504d56a31355d49e1cea5e: Bootstrap starting.
I20260810 06:20:08.413748 1763 tablet_bootstrap.cc:654] T 523f747d5a8348a89c2799324ce905ee P 404f1cff07374d0f8e9a082227bff1c6: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:08.414060 1769 tablet_bootstrap.cc:654] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:08.414127 1768 tablet_bootstrap.cc:654] T 523f747d5a8348a89c2799324ce905ee P 55a32e9e7f504d56a31355d49e1cea5e: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:08.415104 1763 log.cc:824] T 523f747d5a8348a89c2799324ce905ee P 404f1cff07374d0f8e9a082227bff1c6: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:08.415416 1769 log.cc:824] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:08.415452 1768 log.cc:824] T 523f747d5a8348a89c2799324ce905ee P 55a32e9e7f504d56a31355d49e1cea5e: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:08.416980 1763 tablet_bootstrap.cc:492] T 523f747d5a8348a89c2799324ce905ee P 404f1cff07374d0f8e9a082227bff1c6: No bootstrap required, opened a new log
I20260810 06:20:08.417142 1763 ts_tablet_manager.cc:1397] T 523f747d5a8348a89c2799324ce905ee P 404f1cff07374d0f8e9a082227bff1c6: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260810 06:20:08.417205 1768 tablet_bootstrap.cc:492] T 523f747d5a8348a89c2799324ce905ee P 55a32e9e7f504d56a31355d49e1cea5e: No bootstrap required, opened a new log
I20260810 06:20:08.417385 1768 ts_tablet_manager.cc:1397] T 523f747d5a8348a89c2799324ce905ee P 55a32e9e7f504d56a31355d49e1cea5e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260810 06:20:08.419930 1763 raft_consensus.cc:348] T 523f747d5a8348a89c2799324ce905ee P 404f1cff07374d0f8e9a082227bff1c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a32e9e7f504d56a31355d49e1cea5e" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41381 } } peers { permanent_uuid: "93141253bf8b4bf1adb059f3e9db29fb" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46443 } } peers { permanent_uuid: "404f1cff07374d0f8e9a082227bff1c6" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43397 } }
I20260810 06:20:08.420130 1763 raft_consensus.cc:374] T 523f747d5a8348a89c2799324ce905ee P 404f1cff07374d0f8e9a082227bff1c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:08.420183 1763 raft_consensus.cc:729] T 523f747d5a8348a89c2799324ce905ee P 404f1cff07374d0f8e9a082227bff1c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 404f1cff07374d0f8e9a082227bff1c6, State: Initialized, Role: FOLLOWER
I20260810 06:20:08.420183 1768 raft_consensus.cc:348] T 523f747d5a8348a89c2799324ce905ee P 55a32e9e7f504d56a31355d49e1cea5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a32e9e7f504d56a31355d49e1cea5e" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41381 } } peers { permanent_uuid: "93141253bf8b4bf1adb059f3e9db29fb" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46443 } } peers { permanent_uuid: "404f1cff07374d0f8e9a082227bff1c6" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43397 } }
I20260810 06:20:08.420459 1768 raft_consensus.cc:374] T 523f747d5a8348a89c2799324ce905ee P 55a32e9e7f504d56a31355d49e1cea5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:08.420514 1768 raft_consensus.cc:729] T 523f747d5a8348a89c2799324ce905ee P 55a32e9e7f504d56a31355d49e1cea5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a32e9e7f504d56a31355d49e1cea5e, State: Initialized, Role: FOLLOWER
I20260810 06:20:08.420588 1763 consensus_queue.cc:260] T 523f747d5a8348a89c2799324ce905ee P 404f1cff07374d0f8e9a082227bff1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a32e9e7f504d56a31355d49e1cea5e" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41381 } } peers { permanent_uuid: "93141253bf8b4bf1adb059f3e9db29fb" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46443 } } peers { permanent_uuid: "404f1cff07374d0f8e9a082227bff1c6" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43397 } }
I20260810 06:20:08.420884 1768 consensus_queue.cc:260] T 523f747d5a8348a89c2799324ce905ee P 55a32e9e7f504d56a31355d49e1cea5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a32e9e7f504d56a31355d49e1cea5e" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41381 } } peers { permanent_uuid: "93141253bf8b4bf1adb059f3e9db29fb" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46443 } } peers { permanent_uuid: "404f1cff07374d0f8e9a082227bff1c6" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43397 } }
I20260810 06:20:08.421293 1331 heartbeater.cc:503] Master 127.31.204.254:35619 was elected leader, sending a full tablet report...
I20260810 06:20:08.421586 1763 ts_tablet_manager.cc:1428] T 523f747d5a8348a89c2799324ce905ee P 404f1cff07374d0f8e9a082227bff1c6: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20260810 06:20:08.421780 1768 ts_tablet_manager.cc:1428] T 523f747d5a8348a89c2799324ce905ee P 55a32e9e7f504d56a31355d49e1cea5e: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260810 06:20:08.422163 1732 heartbeater.cc:503] Master 127.31.204.254:35619 was elected leader, sending a full tablet report...
I20260810 06:20:08.423158 1769 tablet_bootstrap.cc:492] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb: No bootstrap required, opened a new log
I20260810 06:20:08.423344 1769 ts_tablet_manager.cc:1397] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260810 06:20:08.426076 1769 raft_consensus.cc:348] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a32e9e7f504d56a31355d49e1cea5e" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41381 } } peers { permanent_uuid: "93141253bf8b4bf1adb059f3e9db29fb" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46443 } } peers { permanent_uuid: "404f1cff07374d0f8e9a082227bff1c6" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43397 } }
I20260810 06:20:08.426311 1769 raft_consensus.cc:374] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:08.426455 1769 raft_consensus.cc:729] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93141253bf8b4bf1adb059f3e9db29fb, State: Initialized, Role: FOLLOWER
I20260810 06:20:08.426857 1769 consensus_queue.cc:260] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a32e9e7f504d56a31355d49e1cea5e" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41381 } } peers { permanent_uuid: "93141253bf8b4bf1adb059f3e9db29fb" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46443 } } peers { permanent_uuid: "404f1cff07374d0f8e9a082227bff1c6" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43397 } }
I20260810 06:20:08.427527 1529 heartbeater.cc:503] Master 127.31.204.254:35619 was elected leader, sending a full tablet report...
I20260810 06:20:08.427783 1769 ts_tablet_manager.cc:1428] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260810 06:20:08.446399 1332 tablet.cc:2369] T 523f747d5a8348a89c2799324ce905ee P 404f1cff07374d0f8e9a082227bff1c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:20:08.449045 1779 raft_consensus.cc:482] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:20:08.449182 1779 raft_consensus.cc:504] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a32e9e7f504d56a31355d49e1cea5e" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41381 } } peers { permanent_uuid: "93141253bf8b4bf1adb059f3e9db29fb" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46443 } } peers { permanent_uuid: "404f1cff07374d0f8e9a082227bff1c6" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43397 } }
I20260810 06:20:08.450158 1779 leader_election.cc:288] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55a32e9e7f504d56a31355d49e1cea5e (127.31.204.195:41381), 404f1cff07374d0f8e9a082227bff1c6 (127.31.204.193:43397)
I20260810 06:20:08.453358 1757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:08.450416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:48433 (local address 127.31.204.195:41381)
0810 06:20:08.450528 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:08.450532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:08.450548 (+ 16us) server_negotiation.cc:407] Connection header received
0810 06:20:08.450712 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:08.450715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:08.450756 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:08.450839 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:08.451084 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.451554 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.452499 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.452889 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.452987 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:08.453032 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:08.453192 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 06:20:08.453252 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:20:08.453361 1531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:08.450361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:41381 (local address 127.31.204.194:48433)
0810 06:20:08.450501 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:08.450515 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:08.450590 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:08.450853 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:08.450856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:08.450872 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:08.450986 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:08.450991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.451648 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.451652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:08.452385 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:08.452394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.452988 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.452991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:08.453088 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:08.453105 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 06:20:08.453167 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 06:20:08.453236 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"spinlock_wait_cycles":35712}
I20260810 06:20:08.453977 1761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:08.450637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:40385 (local address 127.31.204.193:43397)
0810 06:20:08.450737 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:08.450741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:08.450879 (+ 138us) server_negotiation.cc:407] Connection header received
0810 06:20:08.451021 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:08.451025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:08.451079 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:08.451148 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:08.451409 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.451880 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.452884 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.453312 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.453433 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:08.453486 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:08.453808 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 06:20:08.453877 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:20:08.453996 1780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:08.450379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:43397 (local address 127.31.204.194:40385)
0810 06:20:08.450785 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:08.450798 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:08.450928 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:08.451165 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:08.451168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:08.451185 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:08.451303 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:08.451309 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.451990 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.451994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:08.452762 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:08.452771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.453549 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.453554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:08.453662 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:08.453689 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 06:20:08.453779 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 06:20:08.453849 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"spinlock_wait_cycles":1152,"thread_start_us":166,"threads_started":1}
I20260810 06:20:08.453948 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "523f747d5a8348a89c2799324ce905ee" candidate_uuid: "93141253bf8b4bf1adb059f3e9db29fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a32e9e7f504d56a31355d49e1cea5e" is_pre_election: true
I20260810 06:20:08.454330 1665 raft_consensus.cc:2454] T 523f747d5a8348a89c2799324ce905ee P 55a32e9e7f504d56a31355d49e1cea5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93141253bf8b4bf1adb059f3e9db29fb in term 0.
I20260810 06:20:08.454535 1244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "523f747d5a8348a89c2799324ce905ee" candidate_uuid: "93141253bf8b4bf1adb059f3e9db29fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404f1cff07374d0f8e9a082227bff1c6" is_pre_election: true
I20260810 06:20:08.454799 1244 raft_consensus.cc:2454] T 523f747d5a8348a89c2799324ce905ee P 404f1cff07374d0f8e9a082227bff1c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93141253bf8b4bf1adb059f3e9db29fb in term 0.
I20260810 06:20:08.454864 1366 leader_election.cc:302] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55a32e9e7f504d56a31355d49e1cea5e, 93141253bf8b4bf1adb059f3e9db29fb; no voters:
I20260810 06:20:08.455116 1779 raft_consensus.cc:2781] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:20:08.455214 1779 raft_consensus.cc:482] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:20:08.455272 1779 raft_consensus.cc:3037] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:08.456283 1779 raft_consensus.cc:504] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a32e9e7f504d56a31355d49e1cea5e" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41381 } } peers { permanent_uuid: "93141253bf8b4bf1adb059f3e9db29fb" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46443 } } peers { permanent_uuid: "404f1cff07374d0f8e9a082227bff1c6" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43397 } }
I20260810 06:20:08.456591 1779 leader_election.cc:288] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb [CANDIDATE]: Term 1 election: Requested vote from peers 55a32e9e7f504d56a31355d49e1cea5e (127.31.204.195:41381), 404f1cff07374d0f8e9a082227bff1c6 (127.31.204.193:43397)
I20260810 06:20:08.456861 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "523f747d5a8348a89c2799324ce905ee" candidate_uuid: "93141253bf8b4bf1adb059f3e9db29fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a32e9e7f504d56a31355d49e1cea5e"
I20260810 06:20:08.456912 1244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "523f747d5a8348a89c2799324ce905ee" candidate_uuid: "93141253bf8b4bf1adb059f3e9db29fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404f1cff07374d0f8e9a082227bff1c6"
I20260810 06:20:08.456972 1665 raft_consensus.cc:3037] T 523f747d5a8348a89c2799324ce905ee P 55a32e9e7f504d56a31355d49e1cea5e [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:08.457019 1244 raft_consensus.cc:3037] T 523f747d5a8348a89c2799324ce905ee P 404f1cff07374d0f8e9a082227bff1c6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:08.457916 1244 raft_consensus.cc:2454] T 523f747d5a8348a89c2799324ce905ee P 404f1cff07374d0f8e9a082227bff1c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93141253bf8b4bf1adb059f3e9db29fb in term 1.
I20260810 06:20:08.458009 1665 raft_consensus.cc:2454] T 523f747d5a8348a89c2799324ce905ee P 55a32e9e7f504d56a31355d49e1cea5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93141253bf8b4bf1adb059f3e9db29fb in term 1.
I20260810 06:20:08.458204 1363 leader_election.cc:302] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 404f1cff07374d0f8e9a082227bff1c6, 93141253bf8b4bf1adb059f3e9db29fb; no voters:
I20260810 06:20:08.458379 1779 raft_consensus.cc:2781] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:08.458590 1779 raft_consensus.cc:686] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb [term 1 LEADER]: Becoming Leader. State: Replica: 93141253bf8b4bf1adb059f3e9db29fb, State: Running, Role: LEADER
I20260810 06:20:08.458871 1779 consensus_queue.cc:237] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a32e9e7f504d56a31355d49e1cea5e" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41381 } } peers { permanent_uuid: "93141253bf8b4bf1adb059f3e9db29fb" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46443 } } peers { permanent_uuid: "404f1cff07374d0f8e9a082227bff1c6" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43397 } }
I20260810 06:20:08.461551 1024 catalog_manager.cc:5495] T 523f747d5a8348a89c2799324ce905ee P 93141253bf8b4bf1adb059f3e9db29fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 93141253bf8b4bf1adb059f3e9db29fb (127.31.204.194). New cstate: current_term: 1 leader_uuid: "93141253bf8b4bf1adb059f3e9db29fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a32e9e7f504d56a31355d49e1cea5e" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93141253bf8b4bf1adb059f3e9db29fb" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "404f1cff07374d0f8e9a082227bff1c6" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43397 } health_report { overall_health: UNKNOWN } } }
Aug 10 06:20:08 dist-test-slave-q9h9 krb5kdc[952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.193@KRBTEST.COM
I20260810 06:20:08.478821 1761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:08.470726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.31.204.193:43397)
0810 06:20:08.470833 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:08.470836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:08.471007 (+ 171us) server_negotiation.cc:407] Connection header received
0810 06:20:08.471107 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:08.471110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:08.471146 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:08.471205 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:08.471831 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.472261 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.473039 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.473231 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.473310 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:08.475927 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:08.475937 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:08.475950 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:08.475972 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:08.477227 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:08.477603 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:08.477609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:08.477611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:08.477700 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:08.477942 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:08.477945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:08.477947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:08.478206 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:08.478379 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:08.478578 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 06:20:08.478678 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Aug 10 06:20:08 dist-test-slave-q9h9 krb5kdc[952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342808, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 06:20:08 dist-test-slave-q9h9 krb5kdc[952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342808, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.204.193@KRBTEST.COM
I20260810 06:20:08.508211 1761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:08.500677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.31.204.193:43397)
0810 06:20:08.500813 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:08.500817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:08.501176 (+ 359us) server_negotiation.cc:407] Connection header received
0810 06:20:08.501298 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:08.501301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:08.501343 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:08.501408 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:08.502076 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.502566 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.503364 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.503579 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.503673 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:08.505764 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:08.505767 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:08.505769 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:08.505792 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:08.506933 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:08.507302 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:08.507305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:08.507306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:08.507379 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:08.507590 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:08.507592 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:08.507593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:08.507736 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:08.507832 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:08.508015 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 06:20:08.508109 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Aug 10 06:20:08 dist-test-slave-q9h9 krb5kdc[952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 06:20:08 dist-test-slave-q9h9 krb5kdc[952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.193@KRBTEST.COM
I20260810 06:20:08.534008 1761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:08.527036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.31.204.193:43397)
0810 06:20:08.527128 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:08.527132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:08.527278 (+ 146us) server_negotiation.cc:407] Connection header received
0810 06:20:08.527390 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:08.527393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:08.527429 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:08.527488 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:08.528079 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.528517 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.529260 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:08.529455 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:08.529534 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:08.531553 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:08.531563 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:08.531565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:08.531588 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:08.532856 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:08.533160 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:08.533162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:08.533163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:08.533233 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:08.533400 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:08.533402 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:08.533403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:08.533540 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:08.533641 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:08.533774 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 06:20:08.533857 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260810 06:20:08.535392 1177 tablet_service.cc:2908] Rejecting scan request for tablet 523f747d5a8348a89c2799324ce905ee: Uninitialized: clean time has not yet been initialized
I20260810 06:20:08.537451 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 1110
I20260810 06:20:08.552459 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 1339
I20260810 06:20:08.570510 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 1535
I20260810 06:20:08.598992 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 970
2026-08-10T06:20:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2356 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:20:08 dist-test-slave-q9h9 krb5kdc[1800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:20:08 dist-test-slave-q9h9 krb5kdc[1800](info): set up 2 sockets
Aug 10 06:20:08 dist-test-slave-q9h9 krb5kdc[1800](info): commencing operation
krb5kdc: starting...
W20260810 06:20:10.099424 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.476s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:20:10 dist-test-slave-q9h9 krb5kdc[1800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:20:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:20:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:20:10.248507 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:35669
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:35669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:10.351395 1815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:10.351603 1815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:10.351655 1815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:10.355476 1815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:10.355520 1815 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 06:20:10.355543 1815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:10.355568 1815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:10.355598 1815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:10.355620 1815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:20:10.360473 1815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:35669
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:35669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:10.361290 1815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:10.362161 1815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:10.369367 1822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:10.369354 1827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:10.369429 1815 server_base.cc:1034] running on GCE node
W20260810 06:20:10.369346 1831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:10.370102 1815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:10.371011 1815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:10.372162 1815 hybrid_clock.cc:648] HybridClock initialized: now 1786342810372137 us; error 44 us; skew 500 ppm
Aug 10 06:20:10 dist-test-slave-q9h9 krb5kdc[1800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342810, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:10.374588 1815 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:20:10.375317 1815 webserver.cc:460] Webserver started at http://127.31.204.254:35647/ using document root <none> and password file <none>
I20260810 06:20:10.375789 1815 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:10.375847 1815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:10.376025 1815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:10.377774 1815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "ebf7fd6bbb2749ebb542950c213682d9"
format_stamp: "Formatted at 2026-08-10 06:20:10 on dist-test-slave-q9h9"
server_key: "e1831ea3c68f57eddafcb77fc86a5b51"
server_key_iv: "eccdcac0e9fdd698aadd8b557045fdd4"
server_key_version: "encryptionkey@0"
I20260810 06:20:10.378279 1815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "ebf7fd6bbb2749ebb542950c213682d9"
format_stamp: "Formatted at 2026-08-10 06:20:10 on dist-test-slave-q9h9"
server_key: "e1831ea3c68f57eddafcb77fc86a5b51"
server_key_iv: "eccdcac0e9fdd698aadd8b557045fdd4"
server_key_version: "encryptionkey@0"
I20260810 06:20:10.381446 1815 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260810 06:20:10.383512 1842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:10.384544 1815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 06:20:10.384642 1815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "ebf7fd6bbb2749ebb542950c213682d9"
format_stamp: "Formatted at 2026-08-10 06:20:10 on dist-test-slave-q9h9"
server_key: "e1831ea3c68f57eddafcb77fc86a5b51"
server_key_iv: "eccdcac0e9fdd698aadd8b557045fdd4"
server_key_version: "encryptionkey@0"
I20260810 06:20:10.384728 1815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:10.399086 1815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:10.401343 1815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:10.401492 1815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:10.408785 1815 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:35669
I20260810 06:20:10.408797 1937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:35669 every 8 connection(s)
I20260810 06:20:10.409551 1815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:20:10.411931 1938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:10.412949 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 1815
I20260810 06:20:10.413069 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:20:10.413345 32563 external_mini_cluster.cc:1428] Setting key cba93489eca57dc7f0d69d55e240717b
I20260810 06:20:10.417512 1938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9: Bootstrap starting.
I20260810 06:20:10.419885 1938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:10.420547 1938 log.cc:824] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9: Log is configured to *not* fsync() on all Append() calls
Aug 10 06:20:10 dist-test-slave-q9h9 krb5kdc[1800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:10.422427 1938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9: No bootstrap required, opened a new log
I20260810 06:20:10.426462 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:10.414586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33266 (local address 127.31.204.254:35669)
0810 06:20:10.414953 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:10.414967 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:10.414999 (+ 32us) server_negotiation.cc:407] Connection header received
0810 06:20:10.415574 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:10.415588 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:10.415876 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:10.416119 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:10.416954 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:10.417872 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:10.418985 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:10.419398 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:10.419500 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:10.421880 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:10.421886 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:10.421900 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:10.421929 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:10.423738 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:10.424129 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:10.424135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:10.424141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:10.424280 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:10.424508 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:10.424511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:10.424513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:10.424954 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:10.425147 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:10.425293 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 06:20:10.425568 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":95,"threads_started":1}
I20260810 06:20:10.426537 1938 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf7fd6bbb2749ebb542950c213682d9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35669 } }
I20260810 06:20:10.426764 1938 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:10.426836 1938 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebf7fd6bbb2749ebb542950c213682d9, State: Initialized, Role: FOLLOWER
I20260810 06:20:10.427203 1938 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf7fd6bbb2749ebb542950c213682d9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35669 } }
I20260810 06:20:10.427316 1938 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:20:10.427373 1938 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:20:10.427492 1938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:10.428234 1938 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf7fd6bbb2749ebb542950c213682d9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35669 } }
I20260810 06:20:10.428534 1938 leader_election.cc:302] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebf7fd6bbb2749ebb542950c213682d9; no voters:
I20260810 06:20:10.428786 1938 leader_election.cc:288] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:20:10.428882 1944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:10.429069 1944 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9 [term 1 LEADER]: Becoming Leader. State: Replica: ebf7fd6bbb2749ebb542950c213682d9, State: Running, Role: LEADER
I20260810 06:20:10.429378 1944 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf7fd6bbb2749ebb542950c213682d9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35669 } }
I20260810 06:20:10.429826 1938 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:10.430836 1945 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebf7fd6bbb2749ebb542950c213682d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf7fd6bbb2749ebb542950c213682d9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35669 } } }
I20260810 06:20:10.430956 1945 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9 [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:10.431109 1946 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebf7fd6bbb2749ebb542950c213682d9. Latest consensus state: current_term: 1 leader_uuid: "ebf7fd6bbb2749ebb542950c213682d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf7fd6bbb2749ebb542950c213682d9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35669 } } }
I20260810 06:20:10.431295 1946 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9 [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:10.431547 1953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:10.434032 1953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:10.438458 1953 catalog_manager.cc:1347] Generated new cluster ID: 646e3db425e844e896d3199f63b28812
I20260810 06:20:10.438519 1953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:10.442881 1953 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:20:10.443614 1953 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:10.449429 1953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9: Generated new TSK 0
I20260810 06:20:10.450071 1953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
I20260810 06:20:10.479971 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35669
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:10.585021 1973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:10.585235 1973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:10.585280 1973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:10.589269 1973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:10.589318 1973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:10.589363 1973 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 06:20:10.589414 1973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:20:10.594388 1973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.204.254:35669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:10.595284 1973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:10.596206 1973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:10.605571 1986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:10.605579 1983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:10.605698 1973 server_base.cc:1034] running on GCE node
W20260810 06:20:10.605583 1982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:10.606446 1973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:10.607120 1973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:10.608307 1973 hybrid_clock.cc:648] HybridClock initialized: now 1786342810608285 us; error 34 us; skew 500 ppm
Aug 10 06:20:10 dist-test-slave-q9h9 krb5kdc[1800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342810, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:10.611027 1973 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:20:10.611919 1973 webserver.cc:460] Webserver started at http://127.31.204.193:44591/ using document root <none> and password file <none>
I20260810 06:20:10.612437 1973 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:10.612493 1973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:10.612659 1973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:10.614650 1973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "09ce2215680048a099eafbdb375fc178"
format_stamp: "Formatted at 2026-08-10 06:20:10 on dist-test-slave-q9h9"
server_key: "9e045f55eeaf6b83c03cf73a14e80ce3"
server_key_iv: "f7370fd2c152a86a54d21f6f921dd364"
server_key_version: "encryptionkey@0"
I20260810 06:20:10.615171 1973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "09ce2215680048a099eafbdb375fc178"
format_stamp: "Formatted at 2026-08-10 06:20:10 on dist-test-slave-q9h9"
server_key: "9e045f55eeaf6b83c03cf73a14e80ce3"
server_key_iv: "f7370fd2c152a86a54d21f6f921dd364"
server_key_version: "encryptionkey@0"
I20260810 06:20:10.618582 1973 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:20:10.620810 1998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:10.621896 1973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:10.622004 1973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "09ce2215680048a099eafbdb375fc178"
format_stamp: "Formatted at 2026-08-10 06:20:10 on dist-test-slave-q9h9"
server_key: "9e045f55eeaf6b83c03cf73a14e80ce3"
server_key_iv: "f7370fd2c152a86a54d21f6f921dd364"
server_key_version: "encryptionkey@0"
I20260810 06:20:10.622128 1973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:10.650938 1973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:10.653723 1973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:10.653884 1973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:10.655272 1973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:10.655318 1973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:10.655354 1973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:10.655380 1973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 06:20:10 dist-test-slave-q9h9 krb5kdc[1800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342810, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:10.665146 1973 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:39849
I20260810 06:20:10.665248 2199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:39849 every 8 connection(s)
I20260810 06:20:10.666100 1973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:20:10.668165 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:10.655991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:36801 (local address 127.31.204.254:35669)
0810 06:20:10.656140 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:10.656145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:10.656942 (+ 797us) server_negotiation.cc:407] Connection header received
0810 06:20:10.658271 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:10.658274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:10.658335 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:10.658430 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:10.659597 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:10.660429 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:10.661811 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:10.662142 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:10.662251 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:10.664897 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:10.664901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:10.664903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:10.664933 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:10.666342 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:10.666880 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:10.666884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:10.666886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:10.666976 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:10.667330 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:10.667332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:10.667333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:10.667491 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:10.667564 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:10.667953 (+ 389us) server_negotiation.cc:299] Negotiation successful
0810 06:20:10.668050 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 06:20:10.668841 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:10.656336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:35669 (local address 127.31.204.193:36801)
0810 06:20:10.656785 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:10.656834 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:10.658078 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:10.658605 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:10.658613 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:10.659003 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:10.659465 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:10.659479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:10.660561 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:10.660565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:10.661665 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:10.661676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:10.662292 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:10.662297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:10.662484 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:10.662998 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:10.663018 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:10.664711 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:10.666446 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:10.666452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:10.666465 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:10.666797 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:10.667071 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:10.667074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:10.667077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:10.667258 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:10.667565 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:10.667569 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:10.667779 (+ 210us) client_negotiation.cc:769] Sending connection context
0810 06:20:10.667919 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 06:20:10.668128 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
I20260810 06:20:10.669922 2200 heartbeater.cc:347] Connected to a master server at 127.31.204.254:35669
I20260810 06:20:10.670078 2200 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:10.670578 2200 heartbeater.cc:511] Master 127.31.204.254:35669 requested a full tablet report, sending...
I20260810 06:20:10.672163 1862 ts_manager.cc:194] Registered new tserver with Master: 09ce2215680048a099eafbdb375fc178 (127.31.204.193:39849)
I20260810 06:20:10.673481 1862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:36801
I20260810 06:20:10.675448 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 1973
I20260810 06:20:10.675558 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:20:10.675824 32563 external_mini_cluster.cc:1428] Setting key b42e757fc48541a9ea16dd103ec226c9
I20260810 06:20:10.678610 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:10.672188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:33379 (local address 127.31.204.254:35669)
0810 06:20:10.672305 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:10.672309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:10.672494 (+ 185us) server_negotiation.cc:407] Connection header received
0810 06:20:10.672598 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:10.672600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:10.672634 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:10.672696 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:10.673110 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:10.673562 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:10.674500 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:10.674709 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:10.674782 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:10.675829 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:10.675833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:10.675836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:10.675861 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:10.677223 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:10.677640 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:10.677642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:10.677643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:10.677703 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:10.677937 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:10.677939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:10.677940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:10.678123 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:10.678194 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:10.678432 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 06:20:10.678511 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 06:20:10.678697 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:10.672123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:35669 (local address 127.31.204.193:33379)
0810 06:20:10.672427 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:10.672441 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:10.672528 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:10.672721 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:10.672724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:10.672915 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:10.673021 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:10.673028 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:10.673684 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:10.673687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:10.674392 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:10.674403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:10.674823 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:10.674827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:10.674914 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:10.675324 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:10.675337 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:10.675707 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:10.677318 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:10.677321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:10.677323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:10.677561 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:10.677770 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:10.677773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:10.677775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:10.677865 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:10.678205 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:10.678207 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:10.678294 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 06:20:10.678401 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 06:20:10.678517 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":1,"thread_start_us":135,"threads_started":1}
WARNING: no policy specified for kudu/127.31.204.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
I20260810 06:20:10.699934 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35669
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:10.805835 2219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:10.806072 2219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:10.806119 2219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:10.809909 2219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:10.809953 2219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:10.809988 2219 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 06:20:10.810050 2219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:20:10.814953 2219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.204.254:35669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:10.815737 2219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:10.816562 2219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:20:10.825048 2219 server_base.cc:1034] running on GCE node
W20260810 06:20:10.825026 2225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:10.826629 2230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:10.826848 2224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:10.827220 2219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:10.827736 2219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:10.828884 2219 hybrid_clock.cc:648] HybridClock initialized: now 1786342810828872 us; error 23 us; skew 500 ppm
Aug 10 06:20:10 dist-test-slave-q9h9 krb5kdc[1800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342810, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:10.831437 2219 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:20:10.832249 2219 webserver.cc:460] Webserver started at http://127.31.204.194:46275/ using document root <none> and password file <none>
I20260810 06:20:10.832705 2219 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:10.832758 2219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:10.832911 2219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:10.834779 2219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "7764270249334db4b5f6bd706bcceab7"
format_stamp: "Formatted at 2026-08-10 06:20:10 on dist-test-slave-q9h9"
server_key: "a7025c29f84e819af729906d792f3fa4"
server_key_iv: "3bfc500e21e08d6b0ca91e09bef00940"
server_key_version: "encryptionkey@0"
I20260810 06:20:10.835280 2219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "7764270249334db4b5f6bd706bcceab7"
format_stamp: "Formatted at 2026-08-10 06:20:10 on dist-test-slave-q9h9"
server_key: "a7025c29f84e819af729906d792f3fa4"
server_key_iv: "3bfc500e21e08d6b0ca91e09bef00940"
server_key_version: "encryptionkey@0"
I20260810 06:20:10.838577 2219 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:20:10.840659 2241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:10.841718 2219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:10.841820 2219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "7764270249334db4b5f6bd706bcceab7"
format_stamp: "Formatted at 2026-08-10 06:20:10 on dist-test-slave-q9h9"
server_key: "a7025c29f84e819af729906d792f3fa4"
server_key_iv: "3bfc500e21e08d6b0ca91e09bef00940"
server_key_version: "encryptionkey@0"
I20260810 06:20:10.841904 2219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:10.866132 2219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:10.868566 2219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:10.868724 2219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:10.870033 2219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:10.870095 2219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:10.870146 2219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:10.870175 2219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 06:20:10 dist-test-slave-q9h9 krb5kdc[1800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342810, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:10.880200 2219 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:36801
I20260810 06:20:10.880261 2439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:36801 every 8 connection(s)
I20260810 06:20:10.880987 2219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:20:10.882416 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:10.870788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:59405 (local address 127.31.204.254:35669)
0810 06:20:10.870926 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:10.870931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:10.872184 (+ 1253us) server_negotiation.cc:407] Connection header received
0810 06:20:10.873169 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:10.873173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:10.873229 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:10.873312 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:10.874817 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:10.875405 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:10.876536 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:10.876753 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:10.876844 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:10.879088 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:10.879091 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:10.879093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:10.879117 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:10.880500 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:10.880947 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:10.880950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:10.880952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:10.881035 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:10.881367 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:10.881370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:10.881372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:10.881591 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:10.881694 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:10.882196 (+ 502us) server_negotiation.cc:299] Negotiation successful
0810 06:20:10.882296 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:20:10.883128 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:10.871614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:35669 (local address 127.31.204.194:59405)
0810 06:20:10.872049 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:10.872096 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:10.872967 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:10.873495 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:10.873503 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:10.873972 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:10.874627 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:10.874648 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:10.875525 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:10.875529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:10.876418 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:10.876428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:10.876898 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:10.876903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:10.877084 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:10.877592 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:10.877606 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:10.878935 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:10.880599 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:10.880604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:10.880617 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:10.880856 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:10.881133 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:10.881136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:10.881138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:10.881274 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:10.881690 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:10.881696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:10.881956 (+ 260us) client_negotiation.cc:769] Sending connection context
0810 06:20:10.882144 (+ 188us) client_negotiation.cc:240] Negotiation successful
0810 06:20:10.882375 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":109,"threads_started":1}
I20260810 06:20:10.884140 2440 heartbeater.cc:347] Connected to a master server at 127.31.204.254:35669
I20260810 06:20:10.884269 2440 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:10.884717 2440 heartbeater.cc:511] Master 127.31.204.254:35669 requested a full tablet report, sending...
I20260810 06:20:10.885610 1864 ts_manager.cc:194] Registered new tserver with Master: 7764270249334db4b5f6bd706bcceab7 (127.31.204.194:36801)
I20260810 06:20:10.886255 1864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:59405
I20260810 06:20:10.887233 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 2219
I20260810 06:20:10.887359 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:20:10.887653 32563 external_mini_cluster.cc:1428] Setting key 8d287603d264abb0dd03ba475305158e
I20260810 06:20:10.892715 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:10.886130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:45865 (local address 127.31.204.254:35669)
0810 06:20:10.886237 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:10.886241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:10.886383 (+ 142us) server_negotiation.cc:407] Connection header received
0810 06:20:10.886562 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:10.886565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:10.886608 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:10.886670 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:10.887106 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:10.887761 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:10.888749 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:10.888995 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:10.889076 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:10.889997 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:10.890000 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:10.890003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:10.890060 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:10.891305 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:10.891674 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:10.891677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:10.891679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:10.891749 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:10.892007 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:10.892010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:10.892011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:10.892205 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:10.892295 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:10.892534 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 06:20:10.892615 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:20:10.892773 2448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:10.886006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:35669 (local address 127.31.204.194:45865)
0810 06:20:10.886300 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:10.886314 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:10.886458 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:10.886713 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:10.886716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:10.886893 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:10.886982 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:10.887005 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:10.887874 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:10.887877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:10.888629 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:10.888640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:10.889089 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:10.889092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:10.889181 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:10.889580 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:10.889593 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:10.889870 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:10.891405 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:10.891408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:10.891410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:10.891589 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:10.891830 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:10.891833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:10.891835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:10.891920 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:10.892303 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:10.892306 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:10.892394 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 06:20:10.892489 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 06:20:10.892607 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.31.204.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
I20260810 06:20:10.925062 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35669
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:11.031821 2453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:11.032028 2453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:11.032073 2453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:11.035961 2453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:11.036011 2453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:11.036046 2453 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 06:20:11.036082 2453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:20:11.040922 2453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.204.254:35669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:11.041725 2453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:11.042608 2453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:11.051079 2461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:11.051216 2462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:11.051107 2468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:11.051321 2453 server_base.cc:1034] running on GCE node
I20260810 06:20:11.051640 2453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:11.052150 2453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:11.053321 2453 hybrid_clock.cc:648] HybridClock initialized: now 1786342811053310 us; error 33 us; skew 500 ppm
Aug 10 06:20:11 dist-test-slave-q9h9 krb5kdc[1800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342811, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:11.055826 2453 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:20:11.056565 2453 webserver.cc:460] Webserver started at http://127.31.204.195:34337/ using document root <none> and password file <none>
I20260810 06:20:11.057029 2453 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:11.057083 2453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:11.057235 2453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:11.059068 2453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "adcaf06e6fca4707afa87b16847f2f56"
format_stamp: "Formatted at 2026-08-10 06:20:11 on dist-test-slave-q9h9"
server_key: "1ab454d845757e60e1998df0fc54a271"
server_key_iv: "b3c156a6027a8f6a5d42ed3fe608614b"
server_key_version: "encryptionkey@0"
I20260810 06:20:11.059531 2453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "adcaf06e6fca4707afa87b16847f2f56"
format_stamp: "Formatted at 2026-08-10 06:20:11 on dist-test-slave-q9h9"
server_key: "1ab454d845757e60e1998df0fc54a271"
server_key_iv: "b3c156a6027a8f6a5d42ed3fe608614b"
server_key_version: "encryptionkey@0"
I20260810 06:20:11.062860 2453 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:20:11.064927 2478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:11.065963 2453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:20:11.066085 2453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "adcaf06e6fca4707afa87b16847f2f56"
format_stamp: "Formatted at 2026-08-10 06:20:11 on dist-test-slave-q9h9"
server_key: "1ab454d845757e60e1998df0fc54a271"
server_key_iv: "b3c156a6027a8f6a5d42ed3fe608614b"
server_key_version: "encryptionkey@0"
I20260810 06:20:11.066169 2453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:11.089689 2453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:11.092221 2453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:11.092353 2453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:11.093618 2453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:11.093668 2453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:11.093717 2453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:11.093745 2453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 06:20:11 dist-test-slave-q9h9 krb5kdc[1800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342811, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:11.105554 2453 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:41205
I20260810 06:20:11.105654 2666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:41205 every 8 connection(s)
I20260810 06:20:11.106499 2453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:20:11.107060 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.094303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:36163 (local address 127.31.204.254:35669)
0810 06:20:11.094416 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:11.094419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:11.095346 (+ 927us) server_negotiation.cc:407] Connection header received
0810 06:20:11.096247 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:11.096251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:11.096309 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:11.096385 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:11.097549 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.098300 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.099456 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.099820 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.099923 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.103270 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:11.103275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:11.103277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:11.103308 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:11.104889 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:11.105467 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:11.105471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:11.105472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:11.105569 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:11.105949 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:11.105953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:11.105955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:11.106212 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:11.106327 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:11.106759 (+ 432us) server_negotiation.cc:299] Negotiation successful
0810 06:20:11.106900 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 06:20:11.107781 2494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.094770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:35669 (local address 127.31.204.195:36163)
0810 06:20:11.095214 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:11.095254 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:11.096045 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:11.096652 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:11.096660 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:11.096985 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:11.097408 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.097421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.098427 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.098431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.099340 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.099350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.099964 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.099969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.100137 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.100650 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:11.100668 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:11.103096 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:11.105001 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:11.105008 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:11.105020 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:11.105364 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:11.105685 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:11.105688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:11.105690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:11.105855 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:11.106317 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:11.106321 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:11.106523 (+ 202us) client_negotiation.cc:769] Sending connection context
0810 06:20:11.106683 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 06:20:11.106959 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":118,"threads_started":1}
I20260810 06:20:11.108826 2667 heartbeater.cc:347] Connected to a master server at 127.31.204.254:35669
I20260810 06:20:11.108959 2667 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:11.109381 2667 heartbeater.cc:511] Master 127.31.204.254:35669 requested a full tablet report, sending...
I20260810 06:20:11.110347 1864 ts_manager.cc:194] Registered new tserver with Master: adcaf06e6fca4707afa87b16847f2f56 (127.31.204.195:41205)
I20260810 06:20:11.110945 1864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:36163
I20260810 06:20:11.111385 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 2453
I20260810 06:20:11.111502 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:20:11.111792 32563 external_mini_cluster.cc:1428] Setting key 309e7ef26f5f544acbb3a7dad67e885b
I20260810 06:20:11.113596 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:20:11.117451 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.110805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:52495 (local address 127.31.204.254:35669)
0810 06:20:11.110927 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:11.110931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:11.111096 (+ 165us) server_negotiation.cc:407] Connection header received
0810 06:20:11.111198 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:11.111201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:11.111243 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:11.111326 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:11.111824 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.112536 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.113424 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.113706 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.113790 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.114780 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:11.114782 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:11.114784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:11.114816 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:11.115949 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:11.116317 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:11.116320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:11.116322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:11.116443 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:11.116690 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:11.116694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:11.116695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:11.116891 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:11.116976 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:11.117212 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 06:20:11.117316 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 06:20:11.117461 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.110733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:35669 (local address 127.31.204.195:52495)
0810 06:20:11.111028 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:11.111041 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:11.111131 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:11.111349 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:11.111352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:11.111586 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:11.111703 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.111709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.112668 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.112672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.113307 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.113316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.113815 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.113818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.113906 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.114321 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:11.114334 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:11.114657 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:11.116049 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:11.116052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:11.116054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:11.116233 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:11.116528 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:11.116531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:11.116533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:11.116613 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:11.116982 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:11.116985 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:11.117068 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 06:20:11.117152 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 06:20:11.117280 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":87,"threads_started":1}
Aug 10 06:20:11 dist-test-slave-q9h9 krb5kdc[1800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342811, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 06:20:11 dist-test-slave-q9h9 krb5kdc[1800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342811, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:11.138414 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.131117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33268 (local address 127.31.204.254:35669)
0810 06:20:11.131246 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:11.131249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:11.131258 (+ 9us) server_negotiation.cc:407] Connection header received
0810 06:20:11.131325 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:11.131327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:11.131365 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:11.131426 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:11.132121 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.132608 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.133400 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.133609 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.133689 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.135816 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:11.135820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:11.135822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:11.135847 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:11.136925 (+ 1078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:11.137299 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:11.137302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:11.137304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:11.137373 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:11.137630 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:11.137633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:11.137635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:11.137792 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:11.137903 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:11.138168 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 06:20:11.138275 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 06:20:11.140731 1864 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:20:11.143631 1864 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:20:11.156921 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.151342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:39849 (local address 127.0.0.1:40808)
0810 06:20:11.151570 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:11.151584 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:11.151668 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:11.153009 (+ 1341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:11.153012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:11.153031 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:11.153127 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.153133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.154516 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.154519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.155726 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.155734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.156449 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.156453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.156555 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.156588 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 06:20:11.156684 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 06:20:11.156787 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":48,"thread_start_us":102,"threads_started":1}
I20260810 06:20:11.157120 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.151927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.31.204.194:36801)
0810 06:20:11.152426 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:11.152437 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:11.152460 (+ 23us) server_negotiation.cc:407] Connection header received
0810 06:20:11.152527 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:11.152532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:11.152720 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:11.153185 (+ 465us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:11.153314 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.154141 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.155758 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.156549 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.156716 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.156816 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:11.156903 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 06:20:11.156989 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":107,"threads_started":1}
I20260810 06:20:11.157135 2697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.151128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:36801 (local address 127.0.0.1:56914)
0810 06:20:11.151461 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:11.151508 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:11.151621 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:11.152895 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:11.152901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:11.152921 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:11.153038 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.153044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.154249 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.154252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.155627 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.155638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.156694 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.156697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.156791 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.156810 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 06:20:11.156878 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 06:20:11.156933 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":58,"thread_start_us":79,"threads_started":1}
I20260810 06:20:11.157507 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.151906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40808 (local address 127.31.204.193:39849)
0810 06:20:11.152544 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:11.152561 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:11.152583 (+ 22us) server_negotiation.cc:407] Connection header received
0810 06:20:11.152658 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:11.152663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:11.152870 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:11.153020 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:11.153260 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.154402 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.155817 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.156339 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.156461 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.156597 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:11.157271 (+ 674us) server_negotiation.cc:299] Negotiation successful
0810 06:20:11.157374 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":62,"threads_started":1}
I20260810 06:20:11.157569 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.152111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34340 (local address 127.31.204.195:41205)
0810 06:20:11.152426 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:11.152437 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:11.152460 (+ 23us) server_negotiation.cc:407] Connection header received
0810 06:20:11.152527 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:11.152532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:11.152720 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:11.152869 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:11.153219 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.154430 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.155786 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.156549 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.156685 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.156817 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:11.157360 (+ 543us) server_negotiation.cc:299] Negotiation successful
0810 06:20:11.157453 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":108,"threads_started":1}
I20260810 06:20:11.157888 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.151830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:41205 (local address 127.0.0.1:34340)
0810 06:20:11.152036 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:11.152050 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:11.152130 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:11.152998 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:11.153001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:11.153016 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:11.153110 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.153115 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.154544 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.154547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.155644 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.155653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.156882 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.156886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.156972 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.157110 (+ 138us) client_negotiation.cc:769] Sending connection context
0810 06:20:11.157182 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 06:20:11.157774 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":60,"spinlock_wait_cycles":13952,"thread_start_us":88,"threads_started":1}
I20260810 06:20:11.159859 2318 tablet_service.cc:1467] Processing CreateTablet for tablet f9d203d7d42d4fe697dc1e6013259287 (DEFAULT_TABLE table=test-table [id=5fb9f9acaf964cb582866ffcdb7ed1a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:11.160148 2088 tablet_service.cc:1467] Processing CreateTablet for tablet f9d203d7d42d4fe697dc1e6013259287 (DEFAULT_TABLE table=test-table [id=5fb9f9acaf964cb582866ffcdb7ed1a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:11.160148 2564 tablet_service.cc:1467] Processing CreateTablet for tablet f9d203d7d42d4fe697dc1e6013259287 (DEFAULT_TABLE table=test-table [id=5fb9f9acaf964cb582866ffcdb7ed1a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:11.160681 2318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9d203d7d42d4fe697dc1e6013259287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:11.161126 2564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9d203d7d42d4fe697dc1e6013259287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:11.161121 2088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9d203d7d42d4fe697dc1e6013259287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:11.167837 2705 tablet_bootstrap.cc:492] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7: Bootstrap starting.
I20260810 06:20:11.168412 2710 tablet_bootstrap.cc:492] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178: Bootstrap starting.
I20260810 06:20:11.168810 2707 tablet_bootstrap.cc:492] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56: Bootstrap starting.
I20260810 06:20:11.170135 2705 tablet_bootstrap.cc:654] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:11.170881 2707 tablet_bootstrap.cc:654] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:11.170959 2710 tablet_bootstrap.cc:654] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:11.171007 2705 log.cc:824] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:11.171609 2707 log.cc:824] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:11.171682 2710 log.cc:824] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:11.173235 2705 tablet_bootstrap.cc:492] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7: No bootstrap required, opened a new log
I20260810 06:20:11.173357 2707 tablet_bootstrap.cc:492] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56: No bootstrap required, opened a new log
I20260810 06:20:11.173429 2705 ts_tablet_manager.cc:1397] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260810 06:20:11.173503 2707 ts_tablet_manager.cc:1397] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 06:20:11.173573 2710 tablet_bootstrap.cc:492] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178: No bootstrap required, opened a new log
I20260810 06:20:11.173741 2710 ts_tablet_manager.cc:1397] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 06:20:11.176405 2710 raft_consensus.cc:348] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.176599 2710 raft_consensus.cc:374] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:11.176651 2710 raft_consensus.cc:729] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ce2215680048a099eafbdb375fc178, State: Initialized, Role: FOLLOWER
I20260810 06:20:11.177022 2710 consensus_queue.cc:260] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.177161 2705 raft_consensus.cc:348] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.177161 2707 raft_consensus.cc:348] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.177402 2705 raft_consensus.cc:374] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:11.177400 2707 raft_consensus.cc:374] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:11.177459 2705 raft_consensus.cc:729] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7764270249334db4b5f6bd706bcceab7, State: Initialized, Role: FOLLOWER
I20260810 06:20:11.177459 2707 raft_consensus.cc:729] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adcaf06e6fca4707afa87b16847f2f56, State: Initialized, Role: FOLLOWER
I20260810 06:20:11.177770 2200 heartbeater.cc:503] Master 127.31.204.254:35669 was elected leader, sending a full tablet report...
I20260810 06:20:11.177884 2705 consensus_queue.cc:260] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.177884 2707 consensus_queue.cc:260] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.177980 2710 ts_tablet_manager.cc:1428] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260810 06:20:11.178562 2440 heartbeater.cc:503] Master 127.31.204.254:35669 was elected leader, sending a full tablet report...
I20260810 06:20:11.178821 2707 ts_tablet_manager.cc:1428] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 06:20:11.179121 2667 heartbeater.cc:503] Master 127.31.204.254:35669 was elected leader, sending a full tablet report...
I20260810 06:20:11.179142 2705 ts_tablet_manager.cc:1428] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260810 06:20:11.357724 2670 tablet.cc:2369] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:20:11.382157 2442 tablet.cc:2369] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:20:11.417362 2201 tablet.cc:2369] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:20:11.454625 2717 raft_consensus.cc:482] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:20:11.454860 2717 raft_consensus.cc:504] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.456139 2717 leader_election.cc:288] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ce2215680048a099eafbdb375fc178 (127.31.204.193:39849), adcaf06e6fca4707afa87b16847f2f56 (127.31.204.195:41205)
I20260810 06:20:11.460466 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.457134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:49009 (local address 127.31.204.195:41205)
0810 06:20:11.457266 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:11.457270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:11.457285 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:20:11.457404 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:11.457406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:11.457452 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:11.457510 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:11.457735 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.458422 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.459581 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.459983 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.460082 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.460136 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:11.460285 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 06:20:11.460348 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:20:11.460485 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.456789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:41205 (local address 127.31.204.194:49009)
0810 06:20:11.457172 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:11.457184 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:11.457278 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:11.457526 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:11.457528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:11.457544 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:11.457640 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.457646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.458513 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.458517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.459458 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.459469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.460082 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.460086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.460176 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.460199 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 06:20:11.460271 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 06:20:11.460339 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":85,"thread_start_us":73,"threads_started":1}
I20260810 06:20:11.461123 2594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d203d7d42d4fe697dc1e6013259287" candidate_uuid: "7764270249334db4b5f6bd706bcceab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adcaf06e6fca4707afa87b16847f2f56" is_pre_election: true
I20260810 06:20:11.461412 2594 raft_consensus.cc:2454] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7764270249334db4b5f6bd706bcceab7 in term 0.
I20260810 06:20:11.461570 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.456517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:43965 (local address 127.31.204.193:39849)
0810 06:20:11.456660 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:11.456665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:11.456714 (+ 49us) server_negotiation.cc:407] Connection header received
0810 06:20:11.456890 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:11.456893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:11.456948 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:11.457056 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:11.457542 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.458366 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.459941 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.460554 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.460648 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.460699 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:11.461379 (+ 680us) server_negotiation.cc:299] Negotiation successful
0810 06:20:11.461460 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 06:20:11.461584 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.456394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:39849 (local address 127.31.204.194:43965)
0810 06:20:11.456627 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:11.456641 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:11.456777 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:11.457206 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:11.457210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:11.457235 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:11.457387 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.457395 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.458522 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.458526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.459785 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.459797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.461089 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.461093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.461214 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.461239 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 06:20:11.461352 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 06:20:11.461428 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":34,"spinlock_wait_cycles":11008,"thread_start_us":90,"threads_started":1}
I20260810 06:20:11.462159 2242 leader_election.cc:302] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7764270249334db4b5f6bd706bcceab7, adcaf06e6fca4707afa87b16847f2f56; no voters:
I20260810 06:20:11.462208 2123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d203d7d42d4fe697dc1e6013259287" candidate_uuid: "7764270249334db4b5f6bd706bcceab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ce2215680048a099eafbdb375fc178" is_pre_election: true
I20260810 06:20:11.462450 2717 raft_consensus.cc:2781] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:20:11.462477 2123 raft_consensus.cc:2454] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7764270249334db4b5f6bd706bcceab7 in term 0.
I20260810 06:20:11.462525 2717 raft_consensus.cc:482] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:20:11.462565 2717 raft_consensus.cc:3037] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:11.463680 2717 raft_consensus.cc:504] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.463958 2717 leader_election.cc:288] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [CANDIDATE]: Term 1 election: Requested vote from peers 09ce2215680048a099eafbdb375fc178 (127.31.204.193:39849), adcaf06e6fca4707afa87b16847f2f56 (127.31.204.195:41205)
I20260810 06:20:11.464319 2594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d203d7d42d4fe697dc1e6013259287" candidate_uuid: "7764270249334db4b5f6bd706bcceab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adcaf06e6fca4707afa87b16847f2f56"
I20260810 06:20:11.464359 2123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d203d7d42d4fe697dc1e6013259287" candidate_uuid: "7764270249334db4b5f6bd706bcceab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ce2215680048a099eafbdb375fc178"
I20260810 06:20:11.464464 2123 raft_consensus.cc:3037] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:11.464438 2594 raft_consensus.cc:3037] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:11.465700 2123 raft_consensus.cc:2454] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7764270249334db4b5f6bd706bcceab7 in term 1.
I20260810 06:20:11.466012 2242 leader_election.cc:302] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ce2215680048a099eafbdb375fc178, 7764270249334db4b5f6bd706bcceab7; no voters:
I20260810 06:20:11.466264 2594 raft_consensus.cc:2454] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7764270249334db4b5f6bd706bcceab7 in term 1.
I20260810 06:20:11.466594 2717 raft_consensus.cc:2781] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:11.466745 2717 raft_consensus.cc:686] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [term 1 LEADER]: Becoming Leader. State: Replica: 7764270249334db4b5f6bd706bcceab7, State: Running, Role: LEADER
I20260810 06:20:11.467024 2717 consensus_queue.cc:237] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.470129 1862 catalog_manager.cc:5495] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7764270249334db4b5f6bd706bcceab7 (127.31.204.194). New cstate: current_term: 1 leader_uuid: "7764270249334db4b5f6bd706bcceab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } health_report { overall_health: UNKNOWN } } }
I20260810 06:20:11.520565 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.517105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.31.204.194:36801)
0810 06:20:11.517224 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:11.517229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:11.517242 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:20:11.517437 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:11.517441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:11.517489 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:11.517569 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:11.517828 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.518340 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.519188 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.519403 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.519488 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.519799 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:11.520191 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:11.520299 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:11.520389 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 06:20:11.520444 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:20:11.526081 2123 raft_consensus.cc:1261] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178 [term 1 FOLLOWER]: Refusing update from remote peer 7764270249334db4b5f6bd706bcceab7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:11.526101 2594 raft_consensus.cc:1261] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56 [term 1 FOLLOWER]: Refusing update from remote peer 7764270249334db4b5f6bd706bcceab7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:11.526734 2717 consensus_queue.cc:1035] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:11.526923 2723 consensus_queue.cc:1035] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:11.536017 2727 mvcc.cc:205] Tried to move back new op lower bound from 7316860156003864576 to 7316860155771371520. Current Snapshot: MvccSnapshot[applied={T|T < 7316860156003864576}]
I20260810 06:20:11.538302 2729 mvcc.cc:205] Tried to move back new op lower bound from 7316860156003864576 to 7316860155771371520. Current Snapshot: MvccSnapshot[applied={T|T < 7316860156003864576 or (T in {7316860156003864576})}]
I20260810 06:20:11.547736 1860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:11.547941 1860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:11.550405 1860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9: Sending DeleteTablet for 3 replicas of tablet f9d203d7d42d4fe697dc1e6013259287
I20260810 06:20:11.551148 2318 tablet_service.cc:1514] Processing DeleteTablet for tablet f9d203d7d42d4fe697dc1e6013259287 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:11 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:56914
I20260810 06:20:11.551291 2088 tablet_service.cc:1514] Processing DeleteTablet for tablet f9d203d7d42d4fe697dc1e6013259287 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:11 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:40808
I20260810 06:20:11.551288 2564 tablet_service.cc:1514] Processing DeleteTablet for tablet f9d203d7d42d4fe697dc1e6013259287 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:11 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:34340
I20260810 06:20:11.551723 2737 tablet_replica.cc:330] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7: stopping tablet replica
I20260810 06:20:11.552052 2737 raft_consensus.cc:2229] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:20:11.552280 2739 tablet_replica.cc:330] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56: stopping tablet replica
I20260810 06:20:11.552395 2737 raft_consensus.cc:2258] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:11.552560 2739 raft_consensus.cc:2229] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:11.552600 1860 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 06:20:11.552829 2739 raft_consensus.cc:2258] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 06:20:11.553005 1860 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:20:11.553181 2738 tablet_replica.cc:330] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178: stopping tablet replica
I20260810 06:20:11.553448 2738 raft_consensus.cc:2229] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:11.553716 2738 raft_consensus.cc:2258] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:11.553753 2737 ts_tablet_manager.cc:1905] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:11.555140 2738 ts_tablet_manager.cc:1905] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:11.556507 2737 ts_tablet_manager.cc:1918] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:20:11.556596 2737 log.cc:1196] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/f9d203d7d42d4fe697dc1e6013259287
I20260810 06:20:11.556900 2737 ts_tablet_manager.cc:1939] T f9d203d7d42d4fe697dc1e6013259287 P 7764270249334db4b5f6bd706bcceab7: Deleting consensus metadata
I20260810 06:20:11.557996 2738 ts_tablet_manager.cc:1918] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:20:11.558001 1847 catalog_manager.cc:4841] TS 7764270249334db4b5f6bd706bcceab7 (127.31.204.194:36801): tablet f9d203d7d42d4fe697dc1e6013259287 (table test-table [id=5fb9f9acaf964cb582866ffcdb7ed1a3]) successfully deleted
I20260810 06:20:11.558130 2738 log.cc:1196] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-0/wal/wals/f9d203d7d42d4fe697dc1e6013259287
I20260810 06:20:11.558470 2738 ts_tablet_manager.cc:1939] T f9d203d7d42d4fe697dc1e6013259287 P 09ce2215680048a099eafbdb375fc178: Deleting consensus metadata
I20260810 06:20:11.559680 1843 catalog_manager.cc:4841] TS 09ce2215680048a099eafbdb375fc178 (127.31.204.193:39849): tablet f9d203d7d42d4fe697dc1e6013259287 (table test-table [id=5fb9f9acaf964cb582866ffcdb7ed1a3]) successfully deleted
I20260810 06:20:11.559907 2088 tablet_service.cc:1467] Processing CreateTablet for tablet 7cde02e548e94c1ebb91803ed07be77b (DEFAULT_TABLE table=test-table [id=9bff549ce1a14a47ae036c1f1b0136e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:11.560012 2739 ts_tablet_manager.cc:1905] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:11.560169 2088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cde02e548e94c1ebb91803ed07be77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:11.560271 2564 tablet_service.cc:1467] Processing CreateTablet for tablet 7cde02e548e94c1ebb91803ed07be77b (DEFAULT_TABLE table=test-table [id=9bff549ce1a14a47ae036c1f1b0136e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:11.560506 2564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cde02e548e94c1ebb91803ed07be77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:11.562430 2710 tablet_bootstrap.cc:492] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178: Bootstrap starting.
I20260810 06:20:11.562506 2739 ts_tablet_manager.cc:1918] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:20:11.562583 2739 log.cc:1196] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/f9d203d7d42d4fe697dc1e6013259287
I20260810 06:20:11.562877 2739 ts_tablet_manager.cc:1939] T f9d203d7d42d4fe697dc1e6013259287 P adcaf06e6fca4707afa87b16847f2f56: Deleting consensus metadata
I20260810 06:20:11.563293 2707 tablet_bootstrap.cc:492] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56: Bootstrap starting.
I20260810 06:20:11.563458 2710 tablet_bootstrap.cc:654] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:11.564298 2707 tablet_bootstrap.cc:654] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:11.564610 2318 tablet_service.cc:1467] Processing CreateTablet for tablet 7cde02e548e94c1ebb91803ed07be77b (DEFAULT_TABLE table=test-table [id=9bff549ce1a14a47ae036c1f1b0136e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:11.564842 2318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cde02e548e94c1ebb91803ed07be77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:11.565003 1843 catalog_manager.cc:4841] TS adcaf06e6fca4707afa87b16847f2f56 (127.31.204.195:41205): tablet f9d203d7d42d4fe697dc1e6013259287 (table test-table [id=5fb9f9acaf964cb582866ffcdb7ed1a3]) successfully deleted
I20260810 06:20:11.565035 2710 tablet_bootstrap.cc:492] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178: No bootstrap required, opened a new log
I20260810 06:20:11.565122 2710 ts_tablet_manager.cc:1397] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 06:20:11.565603 2710 raft_consensus.cc:348] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.565711 2710 raft_consensus.cc:374] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:11.565745 2710 raft_consensus.cc:729] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ce2215680048a099eafbdb375fc178, State: Initialized, Role: FOLLOWER
I20260810 06:20:11.565923 2707 tablet_bootstrap.cc:492] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56: No bootstrap required, opened a new log
I20260810 06:20:11.565867 2710 consensus_queue.cc:260] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.565996 2707 ts_tablet_manager.cc:1397] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 06:20:11.566184 2710 ts_tablet_manager.cc:1428] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:11.566463 2707 raft_consensus.cc:348] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.566591 2707 raft_consensus.cc:374] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:11.566627 2707 raft_consensus.cc:729] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adcaf06e6fca4707afa87b16847f2f56, State: Initialized, Role: FOLLOWER
I20260810 06:20:11.566747 2707 consensus_queue.cc:260] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.567037 2707 ts_tablet_manager.cc:1428] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:11.567993 2705 tablet_bootstrap.cc:492] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7: Bootstrap starting.
I20260810 06:20:11.569039 2705 tablet_bootstrap.cc:654] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:11.570358 2705 tablet_bootstrap.cc:492] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7: No bootstrap required, opened a new log
I20260810 06:20:11.570446 2705 ts_tablet_manager.cc:1397] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260810 06:20:11.570894 2705 raft_consensus.cc:348] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.570998 2705 raft_consensus.cc:374] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:11.571027 2705 raft_consensus.cc:729] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7764270249334db4b5f6bd706bcceab7, State: Initialized, Role: FOLLOWER
I20260810 06:20:11.571141 2705 consensus_queue.cc:260] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.571391 2705 ts_tablet_manager.cc:1428] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:11.741640 2718 raft_consensus.cc:482] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:20:11.741788 2718 raft_consensus.cc:504] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.742817 2718 leader_election.cc:288] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ce2215680048a099eafbdb375fc178 (127.31.204.193:39849), 7764270249334db4b5f6bd706bcceab7 (127.31.204.194:36801)
I20260810 06:20:11.746505 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.743348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:56037 (local address 127.31.204.194:36801)
0810 06:20:11.743474 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:11.743478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:11.743491 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:20:11.743564 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:11.743567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:11.743610 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:11.743670 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:11.743960 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.744483 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.745521 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.745955 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.746086 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.746138 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:11.746322 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 06:20:11.746387 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 06:20:11.746560 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.743120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:48731 (local address 127.31.204.193:39849)
0810 06:20:11.743250 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:11.743255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:11.743379 (+ 124us) server_negotiation.cc:407] Connection header received
0810 06:20:11.743526 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:11.743529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:11.743575 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:11.743635 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:11.743929 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.744418 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.745538 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.745946 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.746082 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.746139 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:11.746373 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 06:20:11.746444 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 06:20:11.746587 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.743019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:39849 (local address 127.31.204.195:48731)
0810 06:20:11.743298 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:11.743312 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:11.743424 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:11.743658 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:11.743660 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:11.743676 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:11.743828 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.743833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.744522 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.744526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.745423 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.745432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.746082 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.746086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.746251 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.746269 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 06:20:11.746335 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 06:20:11.746401 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":61,"spinlock_wait_cycles":22272,"thread_start_us":92,"threads_started":1}
I20260810 06:20:11.746930 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.743141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:36801 (local address 127.31.204.195:56037)
0810 06:20:11.743376 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:11.743389 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:11.743479 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:11.743687 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:11.743690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:11.743732 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:11.743860 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.743865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.744568 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.744572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.745410 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.745419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.746086 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.746090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.746211 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.746229 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 06:20:11.746296 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 06:20:11.746361 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":28,"spinlock_wait_cycles":2560,"thread_start_us":71,"threads_started":1}
I20260810 06:20:11.747056 2123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cde02e548e94c1ebb91803ed07be77b" candidate_uuid: "adcaf06e6fca4707afa87b16847f2f56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ce2215680048a099eafbdb375fc178" is_pre_election: true
I20260810 06:20:11.747206 2123 raft_consensus.cc:2454] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adcaf06e6fca4707afa87b16847f2f56 in term 0.
I20260810 06:20:11.747468 2362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cde02e548e94c1ebb91803ed07be77b" candidate_uuid: "adcaf06e6fca4707afa87b16847f2f56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7764270249334db4b5f6bd706bcceab7" is_pre_election: true
I20260810 06:20:11.747547 2479 leader_election.cc:302] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ce2215680048a099eafbdb375fc178, adcaf06e6fca4707afa87b16847f2f56; no voters:
I20260810 06:20:11.747684 2362 raft_consensus.cc:2454] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adcaf06e6fca4707afa87b16847f2f56 in term 0.
I20260810 06:20:11.747771 2718 raft_consensus.cc:2781] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:20:11.747828 2718 raft_consensus.cc:482] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:20:11.747870 2718 raft_consensus.cc:3037] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:11.748857 2718 raft_consensus.cc:504] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.749135 2718 leader_election.cc:288] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [CANDIDATE]: Term 1 election: Requested vote from peers 09ce2215680048a099eafbdb375fc178 (127.31.204.193:39849), 7764270249334db4b5f6bd706bcceab7 (127.31.204.194:36801)
I20260810 06:20:11.749449 2362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cde02e548e94c1ebb91803ed07be77b" candidate_uuid: "adcaf06e6fca4707afa87b16847f2f56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7764270249334db4b5f6bd706bcceab7"
I20260810 06:20:11.749454 2123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cde02e548e94c1ebb91803ed07be77b" candidate_uuid: "adcaf06e6fca4707afa87b16847f2f56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ce2215680048a099eafbdb375fc178"
I20260810 06:20:11.749547 2123 raft_consensus.cc:3037] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:11.749549 2362 raft_consensus.cc:3037] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:11.750428 2123 raft_consensus.cc:2454] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adcaf06e6fca4707afa87b16847f2f56 in term 1.
I20260810 06:20:11.750610 2362 raft_consensus.cc:2454] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adcaf06e6fca4707afa87b16847f2f56 in term 1.
I20260810 06:20:11.750909 2483 leader_election.cc:302] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7764270249334db4b5f6bd706bcceab7, adcaf06e6fca4707afa87b16847f2f56; no voters:
I20260810 06:20:11.751060 2718 raft_consensus.cc:2781] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:11.751214 2718 raft_consensus.cc:686] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [term 1 LEADER]: Becoming Leader. State: Replica: adcaf06e6fca4707afa87b16847f2f56, State: Running, Role: LEADER
I20260810 06:20:11.751447 2718 consensus_queue.cc:237] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.753324 1860 catalog_manager.cc:5495] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 reported cstate change: term changed from 0 to 1, leader changed from <none> to adcaf06e6fca4707afa87b16847f2f56 (127.31.204.195). New cstate: current_term: 1 leader_uuid: "adcaf06e6fca4707afa87b16847f2f56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } health_report { overall_health: HEALTHY } } }
Aug 10 06:20:11 dist-test-slave-q9h9 krb5kdc[1800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342810, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:11.826912 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.819444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:35669 (local address 127.0.0.1:33280)
0810 06:20:11.819718 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:11.819730 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:11.819825 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:11.820025 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:11.820027 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:11.820406 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:11.820503 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.820509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.821177 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.821179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.821722 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.821728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.822144 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.822147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.822206 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.822696 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:11.822707 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:11.824023 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:11.825570 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:11.825577 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:11.825582 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:11.825795 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:11.826000 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:11.826002 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:11.826003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:11.826118 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:11.826432 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:11.826437 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:11.826532 (+ 95us) client_negotiation.cc:769] Sending connection context
0810 06:20:11.826618 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 06:20:11.826727 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":96,"threads_started":1}
I20260810 06:20:11.826921 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.819513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.31.204.254:35669)
0810 06:20:11.819745 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:11.819752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:11.819781 (+ 29us) server_negotiation.cc:407] Connection header received
0810 06:20:11.819890 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:11.819892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:11.819929 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:11.820001 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:11.820586 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.821105 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.821796 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.822006 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.822129 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.824156 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:11.824160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:11.824162 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:11.824199 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:11.825490 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:11.825865 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:11.825867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:11.825868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:11.825937 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:11.826184 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:11.826187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:11.826197 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:11.826353 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:11.826439 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:11.826664 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 06:20:11.826784 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":171,"thread_start_us":88,"threads_started":1}
I20260810 06:20:11.830428 1860 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:33280:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 06:20:11.831130 1860 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:20:11.835490 2088 tablet_service.cc:1467] Processing CreateTablet for tablet 81243af1318647a29f6d2f6eafa05e96 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=55281b76cd844deea0d2248dd182389a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 06:20:11.835587 2318 tablet_service.cc:1467] Processing CreateTablet for tablet 81243af1318647a29f6d2f6eafa05e96 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=55281b76cd844deea0d2248dd182389a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 06:20:11.835927 2088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81243af1318647a29f6d2f6eafa05e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:11.835963 2318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81243af1318647a29f6d2f6eafa05e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:11.838404 2710 tablet_bootstrap.cc:492] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178: Bootstrap starting.
I20260810 06:20:11.838578 2564 tablet_service.cc:1467] Processing CreateTablet for tablet 81243af1318647a29f6d2f6eafa05e96 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=55281b76cd844deea0d2248dd182389a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 06:20:11.838896 2564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81243af1318647a29f6d2f6eafa05e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:11.839246 2710 tablet_bootstrap.cc:654] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:11.840428 2710 tablet_bootstrap.cc:492] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178: No bootstrap required, opened a new log
I20260810 06:20:11.840517 2710 ts_tablet_manager.cc:1397] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:11.840960 2710 raft_consensus.cc:348] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.841076 2710 raft_consensus.cc:374] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:11.841118 2710 raft_consensus.cc:729] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ce2215680048a099eafbdb375fc178, State: Initialized, Role: FOLLOWER
I20260810 06:20:11.841248 2710 consensus_queue.cc:260] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.841413 2707 tablet_bootstrap.cc:492] T 81243af1318647a29f6d2f6eafa05e96 P adcaf06e6fca4707afa87b16847f2f56: Bootstrap starting.
I20260810 06:20:11.841483 2705 tablet_bootstrap.cc:492] T 81243af1318647a29f6d2f6eafa05e96 P 7764270249334db4b5f6bd706bcceab7: Bootstrap starting.
I20260810 06:20:11.841516 2710 ts_tablet_manager.cc:1428] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:11.841777 2716 tablet_replica.cc:439] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } } }
I20260810 06:20:11.841917 2716 tablet_replica.cc:442] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:11.842357 2705 tablet_bootstrap.cc:654] T 81243af1318647a29f6d2f6eafa05e96 P 7764270249334db4b5f6bd706bcceab7: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:11.842373 2707 tablet_bootstrap.cc:654] T 81243af1318647a29f6d2f6eafa05e96 P adcaf06e6fca4707afa87b16847f2f56: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:11.843516 2707 tablet_bootstrap.cc:492] T 81243af1318647a29f6d2f6eafa05e96 P adcaf06e6fca4707afa87b16847f2f56: No bootstrap required, opened a new log
I20260810 06:20:11.843597 2707 ts_tablet_manager.cc:1397] T 81243af1318647a29f6d2f6eafa05e96 P adcaf06e6fca4707afa87b16847f2f56: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:11.843958 2705 tablet_bootstrap.cc:492] T 81243af1318647a29f6d2f6eafa05e96 P 7764270249334db4b5f6bd706bcceab7: No bootstrap required, opened a new log
I20260810 06:20:11.843986 2707 raft_consensus.cc:348] T 81243af1318647a29f6d2f6eafa05e96 P adcaf06e6fca4707afa87b16847f2f56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.844048 2705 ts_tablet_manager.cc:1397] T 81243af1318647a29f6d2f6eafa05e96 P 7764270249334db4b5f6bd706bcceab7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 06:20:11.844086 2707 raft_consensus.cc:374] T 81243af1318647a29f6d2f6eafa05e96 P adcaf06e6fca4707afa87b16847f2f56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:11.844121 2707 raft_consensus.cc:729] T 81243af1318647a29f6d2f6eafa05e96 P adcaf06e6fca4707afa87b16847f2f56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adcaf06e6fca4707afa87b16847f2f56, State: Initialized, Role: FOLLOWER
I20260810 06:20:11.844235 2707 consensus_queue.cc:260] T 81243af1318647a29f6d2f6eafa05e96 P adcaf06e6fca4707afa87b16847f2f56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.844429 2705 raft_consensus.cc:348] T 81243af1318647a29f6d2f6eafa05e96 P 7764270249334db4b5f6bd706bcceab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.844501 2707 ts_tablet_manager.cc:1428] T 81243af1318647a29f6d2f6eafa05e96 P adcaf06e6fca4707afa87b16847f2f56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:11.844534 2705 raft_consensus.cc:374] T 81243af1318647a29f6d2f6eafa05e96 P 7764270249334db4b5f6bd706bcceab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:11.844573 2705 raft_consensus.cc:729] T 81243af1318647a29f6d2f6eafa05e96 P 7764270249334db4b5f6bd706bcceab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7764270249334db4b5f6bd706bcceab7, State: Initialized, Role: FOLLOWER
I20260810 06:20:11.844671 2718 tablet_replica.cc:439] T 81243af1318647a29f6d2f6eafa05e96 P adcaf06e6fca4707afa87b16847f2f56: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } } }
I20260810 06:20:11.844697 2705 consensus_queue.cc:260] T 81243af1318647a29f6d2f6eafa05e96 P 7764270249334db4b5f6bd706bcceab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.844796 2718 tablet_replica.cc:442] T 81243af1318647a29f6d2f6eafa05e96 P adcaf06e6fca4707afa87b16847f2f56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:11.844935 2705 ts_tablet_manager.cc:1428] T 81243af1318647a29f6d2f6eafa05e96 P 7764270249334db4b5f6bd706bcceab7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:11.845112 2732 tablet_replica.cc:439] T 81243af1318647a29f6d2f6eafa05e96 P 7764270249334db4b5f6bd706bcceab7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } } }
I20260810 06:20:11.845238 2732 tablet_replica.cc:442] T 81243af1318647a29f6d2f6eafa05e96 P 7764270249334db4b5f6bd706bcceab7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:11.887282 2716 raft_consensus.cc:482] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:20:11.887398 2716 raft_consensus.cc:504] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.888545 2716 leader_election.cc:288] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7764270249334db4b5f6bd706bcceab7 (127.31.204.194:36801), adcaf06e6fca4707afa87b16847f2f56 (127.31.204.195:41205)
I20260810 06:20:11.892994 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.888880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:60219 (local address 127.31.204.194:36801)
0810 06:20:11.889075 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:11.889078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:11.889269 (+ 191us) server_negotiation.cc:407] Connection header received
0810 06:20:11.889394 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:11.889397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:11.889440 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:11.889514 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:11.889795 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.890588 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.892023 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.892410 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.892514 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.892566 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:11.892813 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 06:20:11.892889 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260810 06:20:11.893030 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.888748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:36801 (local address 127.31.204.193:60219)
0810 06:20:11.889133 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:11.889148 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:11.889305 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:11.889532 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:11.889536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:11.889553 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:11.889700 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.889707 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.890711 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.890714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.891876 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.891882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.892537 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.892541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.892648 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.892673 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 06:20:11.892790 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 06:20:11.892863 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":31,"spinlock_wait_cycles":13056,"thread_start_us":130,"threads_started":1}
I20260810 06:20:11.893108 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.888873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:44343 (local address 127.31.204.195:41205)
0810 06:20:11.888995 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:11.888999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:11.889315 (+ 316us) server_negotiation.cc:407] Connection header received
0810 06:20:11.889488 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:11.889492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:11.889544 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:11.889609 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:11.889925 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.890440 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.892002 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.892605 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.892711 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.892754 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:11.892931 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 06:20:11.893006 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:20:11.893424 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.888764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:41205 (local address 127.31.204.193:44343)
0810 06:20:11.889250 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:11.889262 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:11.889338 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:11.889702 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:11.889705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:11.889722 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:11.889820 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.889826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.890546 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.890550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.891866 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.891878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.892710 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.892714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.892804 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.892822 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 06:20:11.893240 (+ 418us) client_negotiation.cc:240] Negotiation successful
0810 06:20:11.893316 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":3,"thread_start_us":114,"threads_started":1}
I20260810 06:20:11.893524 2362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81243af1318647a29f6d2f6eafa05e96" candidate_uuid: "09ce2215680048a099eafbdb375fc178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7764270249334db4b5f6bd706bcceab7" is_pre_election: true
I20260810 06:20:11.893647 2362 raft_consensus.cc:2454] T 81243af1318647a29f6d2f6eafa05e96 P 7764270249334db4b5f6bd706bcceab7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ce2215680048a099eafbdb375fc178 in term 0.
I20260810 06:20:11.893806 2594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81243af1318647a29f6d2f6eafa05e96" candidate_uuid: "09ce2215680048a099eafbdb375fc178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adcaf06e6fca4707afa87b16847f2f56" is_pre_election: true
I20260810 06:20:11.893935 2594 raft_consensus.cc:2454] T 81243af1318647a29f6d2f6eafa05e96 P adcaf06e6fca4707afa87b16847f2f56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ce2215680048a099eafbdb375fc178 in term 0.
I20260810 06:20:11.893978 2003 leader_election.cc:302] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ce2215680048a099eafbdb375fc178, 7764270249334db4b5f6bd706bcceab7; no voters:
I20260810 06:20:11.894274 2716 raft_consensus.cc:2781] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:20:11.894340 2716 raft_consensus.cc:482] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:20:11.894373 2716 raft_consensus.cc:3037] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:11.895543 2716 raft_consensus.cc:504] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.895875 2716 leader_election.cc:288] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 [CANDIDATE]: Term 1 election: Requested vote from peers 7764270249334db4b5f6bd706bcceab7 (127.31.204.194:36801), adcaf06e6fca4707afa87b16847f2f56 (127.31.204.195:41205)
I20260810 06:20:11.896232 2362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81243af1318647a29f6d2f6eafa05e96" candidate_uuid: "09ce2215680048a099eafbdb375fc178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7764270249334db4b5f6bd706bcceab7"
I20260810 06:20:11.896342 2362 raft_consensus.cc:3037] T 81243af1318647a29f6d2f6eafa05e96 P 7764270249334db4b5f6bd706bcceab7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:11.896317 2594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81243af1318647a29f6d2f6eafa05e96" candidate_uuid: "09ce2215680048a099eafbdb375fc178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adcaf06e6fca4707afa87b16847f2f56"
I20260810 06:20:11.896420 2594 raft_consensus.cc:3037] T 81243af1318647a29f6d2f6eafa05e96 P adcaf06e6fca4707afa87b16847f2f56 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:11.897192 2362 raft_consensus.cc:2454] T 81243af1318647a29f6d2f6eafa05e96 P 7764270249334db4b5f6bd706bcceab7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ce2215680048a099eafbdb375fc178 in term 1.
I20260810 06:20:11.897431 2594 raft_consensus.cc:2454] T 81243af1318647a29f6d2f6eafa05e96 P adcaf06e6fca4707afa87b16847f2f56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ce2215680048a099eafbdb375fc178 in term 1.
I20260810 06:20:11.897433 2003 leader_election.cc:302] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ce2215680048a099eafbdb375fc178, 7764270249334db4b5f6bd706bcceab7; no voters:
I20260810 06:20:11.897655 2716 raft_consensus.cc:2781] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:11.897884 2716 raft_consensus.cc:686] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 [term 1 LEADER]: Becoming Leader. State: Replica: 09ce2215680048a099eafbdb375fc178, State: Running, Role: LEADER
I20260810 06:20:11.898145 2716 consensus_queue.cc:237] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } }
I20260810 06:20:11.899572 2773 tablet_replica.cc:439] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178: TxnStatusTablet state changed. Reason: New leader 09ce2215680048a099eafbdb375fc178. Latest consensus state: current_term: 1 leader_uuid: "09ce2215680048a099eafbdb375fc178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } } }
I20260810 06:20:11.899679 2773 tablet_replica.cc:442] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178: This TxnStatusTablet replica's current role is: LEADER
I20260810 06:20:11.900101 2775 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 06:20:11.900302 2775 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 06:20:11.900331 1864 catalog_manager.cc:5495] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ce2215680048a099eafbdb375fc178 (127.31.204.193). New cstate: current_term: 1 leader_uuid: "09ce2215680048a099eafbdb375fc178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } health_report { overall_health: UNKNOWN } } }
I20260810 06:20:11.921289 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.917687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40814 (local address 127.31.204.193:39849)
0810 06:20:11.917789 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:11.917794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:11.917820 (+ 26us) server_negotiation.cc:407] Connection header received
0810 06:20:11.917989 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:11.917991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:11.918087 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:11.918183 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:11.918442 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.918922 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.920057 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.920255 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.920322 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.920614 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:11.920975 (+ 361us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:11.921053 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:11.921152 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 06:20:11.921197 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:20:11.921352 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.917606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:39849 (local address 127.0.0.1:40814)
0810 06:20:11.917747 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:11.917773 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:11.917866 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:11.918212 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:11.918215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:11.918224 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:20:11.918330 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.918336 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.919008 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.919011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.919960 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.919970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.920346 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.920350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.920445 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.920497 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:20:11.921054 (+ 557us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:20:11.921067 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 06:20:11.921132 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 06:20:11.921221 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 06:20:11.923882 2594 raft_consensus.cc:1261] T 81243af1318647a29f6d2f6eafa05e96 P adcaf06e6fca4707afa87b16847f2f56 [term 1 FOLLOWER]: Refusing update from remote peer 09ce2215680048a099eafbdb375fc178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:11.923988 2362 raft_consensus.cc:1261] T 81243af1318647a29f6d2f6eafa05e96 P 7764270249334db4b5f6bd706bcceab7 [term 1 FOLLOWER]: Refusing update from remote peer 09ce2215680048a099eafbdb375fc178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:11.924332 2773 consensus_queue.cc:1035] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:11.924508 2716 consensus_queue.cc:1035] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:11.926040 2732 tablet_replica.cc:439] T 81243af1318647a29f6d2f6eafa05e96 P 7764270249334db4b5f6bd706bcceab7: TxnStatusTablet state changed. Reason: New leader 09ce2215680048a099eafbdb375fc178. Latest consensus state: current_term: 1 leader_uuid: "09ce2215680048a099eafbdb375fc178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } } }
I20260810 06:20:11.926151 2732 tablet_replica.cc:442] T 81243af1318647a29f6d2f6eafa05e96 P 7764270249334db4b5f6bd706bcceab7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:11.926904 2718 tablet_replica.cc:439] T 81243af1318647a29f6d2f6eafa05e96 P adcaf06e6fca4707afa87b16847f2f56: TxnStatusTablet state changed. Reason: New leader 09ce2215680048a099eafbdb375fc178. Latest consensus state: current_term: 1 leader_uuid: "09ce2215680048a099eafbdb375fc178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } } }
I20260810 06:20:11.927021 2718 tablet_replica.cc:442] T 81243af1318647a29f6d2f6eafa05e96 P adcaf06e6fca4707afa87b16847f2f56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:11.928282 2716 tablet_replica.cc:439] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09ce2215680048a099eafbdb375fc178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } } }
I20260810 06:20:11.928386 2716 tablet_replica.cc:442] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178: This TxnStatusTablet replica's current role is: LEADER
I20260810 06:20:11.929188 2718 tablet_replica.cc:439] T 81243af1318647a29f6d2f6eafa05e96 P adcaf06e6fca4707afa87b16847f2f56: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09ce2215680048a099eafbdb375fc178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } } }
I20260810 06:20:11.929292 2718 tablet_replica.cc:442] T 81243af1318647a29f6d2f6eafa05e96 P adcaf06e6fca4707afa87b16847f2f56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:11.929494 2716 tablet_replica.cc:439] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09ce2215680048a099eafbdb375fc178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } } }
I20260810 06:20:11.929589 2716 tablet_replica.cc:442] T 81243af1318647a29f6d2f6eafa05e96 P 09ce2215680048a099eafbdb375fc178: This TxnStatusTablet replica's current role is: LEADER
I20260810 06:20:11.930678 2726 mvcc.cc:205] Tried to move back new op lower bound from 7316860157635858432 to 7316860157536817152. Current Snapshot: MvccSnapshot[applied={T|T < 7316860157635858432}]
I20260810 06:20:11.930886 2732 tablet_replica.cc:439] T 81243af1318647a29f6d2f6eafa05e96 P 7764270249334db4b5f6bd706bcceab7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09ce2215680048a099eafbdb375fc178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 } } peers { permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 } } peers { permanent_uuid: "adcaf06e6fca4707afa87b16847f2f56" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 41205 } } }
I20260810 06:20:11.930971 2732 tablet_replica.cc:442] T 81243af1318647a29f6d2f6eafa05e96 P 7764270249334db4b5f6bd706bcceab7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:11.937084 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.933855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.31.204.195:41205)
0810 06:20:11.933952 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:11.933956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:11.934034 (+ 78us) server_negotiation.cc:407] Connection header received
0810 06:20:11.934221 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:11.934224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:11.934279 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:11.934355 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:11.934581 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.935059 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.935833 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.936048 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.936129 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.936382 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:11.936731 (+ 349us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:11.936828 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:11.936901 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 06:20:11.936947 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 06:20:11.946946 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.943792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:49121 (local address 127.31.204.193:39849)
0810 06:20:11.943925 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:11.943930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:11.944199 (+ 269us) server_negotiation.cc:407] Connection header received
0810 06:20:11.944332 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:11.944334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:11.944377 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:11.944433 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:11.944655 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.945116 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.945956 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.946228 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.946315 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.946580 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:11.946643 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:11.946698 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:11.946791 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 06:20:11.946833 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 06:20:11.946982 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.943654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:39849 (local address 127.31.204.195:49121)
0810 06:20:11.944127 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:11.944141 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:11.944239 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:11.944455 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:11.944458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:11.944465 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:20:11.944566 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.944571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.945215 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.945219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.945850 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.945858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.946328 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.946332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.946424 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.946461 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:20:11.946712 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:20:11.946719 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 06:20:11.946775 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 06:20:11.946848 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":308,"threads_started":1}
I20260810 06:20:11.952754 2123 raft_consensus.cc:1261] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178 [term 1 FOLLOWER]: Refusing update from remote peer adcaf06e6fca4707afa87b16847f2f56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:11.952802 2362 raft_consensus.cc:1261] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7 [term 1 FOLLOWER]: Refusing update from remote peer adcaf06e6fca4707afa87b16847f2f56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:11.953248 2749 consensus_queue.cc:1035] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7764270249334db4b5f6bd706bcceab7" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:11.953567 2718 consensus_queue.cc:1035] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ce2215680048a099eafbdb375fc178" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:11.974926 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.971256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:42625 (local address 127.31.204.195:41205)
0810 06:20:11.971349 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:11.971353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:11.971635 (+ 282us) server_negotiation.cc:407] Connection header received
0810 06:20:11.971758 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:11.971761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:11.971800 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:11.971851 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:11.972075 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.972535 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.973883 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.974212 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.974324 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.974590 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:11.974648 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:11.974693 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:11.974792 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 06:20:11.974839 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 06:20:11.975258 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:11.970934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:41205 (local address 127.31.204.193:42625)
0810 06:20:11.971550 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:11.971566 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:11.971667 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:11.971871 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:11.971874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:11.971881 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:20:11.971998 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.972004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.972616 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.972619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.973773 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:11.973785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:11.974325 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:11.974328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:11.974424 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:11.974475 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:20:11.974718 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:20:11.974724 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 06:20:11.974786 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 06:20:11.974867 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":88,"threads_started":1}
I20260810 06:20:11.997462 1860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:11.997594 1860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:11.999439 1860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebf7fd6bbb2749ebb542950c213682d9: Sending DeleteTablet for 3 replicas of tablet 7cde02e548e94c1ebb91803ed07be77b
I20260810 06:20:11.999969 2088 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cde02e548e94c1ebb91803ed07be77b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:11 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:40808
I20260810 06:20:12.000099 2318 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cde02e548e94c1ebb91803ed07be77b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:11 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:56914
I20260810 06:20:12.000247 2738 tablet_replica.cc:330] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178: stopping tablet replica
I20260810 06:20:12.000301 2737 tablet_replica.cc:330] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7: stopping tablet replica
I20260810 06:20:12.000360 2738 raft_consensus.cc:2229] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:12.000415 2737 raft_consensus.cc:2229] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:12.000468 2738 raft_consensus.cc:2258] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:12.000537 2737 raft_consensus.cc:2258] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:12.002345 2738 ts_tablet_manager.cc:1905] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:12.004586 2738 ts_tablet_manager.cc:1918] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:20:12.004652 2738 log.cc:1196] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-0/wal/wals/7cde02e548e94c1ebb91803ed07be77b
I20260810 06:20:12.004876 2738 ts_tablet_manager.cc:1939] T 7cde02e548e94c1ebb91803ed07be77b P 09ce2215680048a099eafbdb375fc178: Deleting consensus metadata
I20260810 06:20:12.005486 1843 catalog_manager.cc:4841] TS 09ce2215680048a099eafbdb375fc178 (127.31.204.193:39849): tablet 7cde02e548e94c1ebb91803ed07be77b (table test-table [id=9bff549ce1a14a47ae036c1f1b0136e1]) successfully deleted
I20260810 06:20:12.005729 2737 ts_tablet_manager.cc:1905] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:12.006268 2564 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cde02e548e94c1ebb91803ed07be77b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:11 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:34340
I20260810 06:20:12.006448 2739 tablet_replica.cc:330] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56: stopping tablet replica
I20260810 06:20:12.006554 2739 raft_consensus.cc:2229] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:20:12.006829 2739 raft_consensus.cc:2258] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:12.008890 2739 ts_tablet_manager.cc:1905] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:12.011615 2739 ts_tablet_manager.cc:1918] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:20:12.011696 2739 log.cc:1196] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/7cde02e548e94c1ebb91803ed07be77b
I20260810 06:20:12.011981 2739 ts_tablet_manager.cc:1939] T 7cde02e548e94c1ebb91803ed07be77b P adcaf06e6fca4707afa87b16847f2f56: Deleting consensus metadata
I20260810 06:20:12.012629 1843 catalog_manager.cc:4841] TS adcaf06e6fca4707afa87b16847f2f56 (127.31.204.195:41205): tablet 7cde02e548e94c1ebb91803ed07be77b (table test-table [id=9bff549ce1a14a47ae036c1f1b0136e1]) successfully deleted
I20260810 06:20:12.013005 2737 ts_tablet_manager.cc:1918] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:20:12.013073 2737 log.cc:1196] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/7cde02e548e94c1ebb91803ed07be77b
I20260810 06:20:12.013298 2737 ts_tablet_manager.cc:1939] T 7cde02e548e94c1ebb91803ed07be77b P 7764270249334db4b5f6bd706bcceab7: Deleting consensus metadata
Aug 10 06:20:12 dist-test-slave-q9h9 krb5kdc[1800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342811, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.204.193@KRBTEST.COM
I20260810 06:20:12.013888 1847 catalog_manager.cc:4841] TS 7764270249334db4b5f6bd706bcceab7 (127.31.204.194:36801): tablet 7cde02e548e94c1ebb91803ed07be77b (table test-table [id=9bff549ce1a14a47ae036c1f1b0136e1]) successfully deleted
I20260810 06:20:12.017860 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:12.001714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40818 (local address 127.31.204.193:39849)
0810 06:20:12.001816 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:12.001821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:12.006539 (+ 4718us) server_negotiation.cc:407] Connection header received
0810 06:20:12.006772 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:12.006776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:12.006824 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:12.006873 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:12.007861 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:12.008548 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:12.009723 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:12.010046 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:12.010120 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:12.014662 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:12.014673 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:12.014687 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:12.014709 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:12.016227 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:12.016610 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:12.016615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:12.016616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:12.016705 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:12.016949 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:12.016952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:12.016954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:12.017171 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:12.017367 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:12.017565 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 06:20:12.017670 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260810 06:20:12.018384 2192 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40818
I20260810 06:20:12.026777 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:12.019793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.31.204.254:35669)
0810 06:20:12.019922 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:12.019927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:12.020231 (+ 304us) server_negotiation.cc:407] Connection header received
0810 06:20:12.020322 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:12.020325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:12.020369 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:12.020431 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:12.021007 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:12.021524 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:12.022304 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:12.022537 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:12.022620 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:12.023862 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:12.023866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:12.023868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:12.023893 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:12.025143 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:12.025545 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:12.025549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:12.025551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:12.025625 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:12.025870 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:12.025873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:12.025875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:12.026058 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:12.026176 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:12.026447 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 06:20:12.026631 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 06:20:12.027066 1860 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33282
I20260810 06:20:12.028371 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 1973
I20260810 06:20:12.040102 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 2219
I20260810 06:20:12.057967 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 2453
I20260810 06:20:12.066075 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 1815
2026-08-10T06:20:12Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3469 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:20:12 dist-test-slave-q9h9 krb5kdc[2821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:20:12 dist-test-slave-q9h9 krb5kdc[2821](info): set up 2 sockets
Aug 10 06:20:12 dist-test-slave-q9h9 krb5kdc[2821](info): commencing operation
krb5kdc: starting...
W20260810 06:20:13.224974 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:20:13 dist-test-slave-q9h9 krb5kdc[2821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:20:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:20:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:20:13.333645 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:41717
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:38195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:41717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:13.436980 2836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:13.437206 2836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:13.437258 2836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:13.441094 2836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:13.441140 2836 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 06:20:13.441164 2836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:13.441185 2836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:13.441207 2836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:13.441236 2836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:20:13.446146 2836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:38195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:41717
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:41717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:13.447026 2836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:13.447913 2836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:13.455575 2845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:13.455581 2843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:13.455602 2849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:13.455848 2836 server_base.cc:1034] running on GCE node
I20260810 06:20:13.456355 2836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:13.457300 2836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:13.458451 2836 hybrid_clock.cc:648] HybridClock initialized: now 1786342813458441 us; error 28 us; skew 500 ppm
Aug 10 06:20:13 dist-test-slave-q9h9 krb5kdc[2821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342813, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:13.465055 2836 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:20:13.465869 2836 webserver.cc:460] Webserver started at http://127.31.204.254:42281/ using document root <none> and password file <none>
I20260810 06:20:13.466375 2836 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:13.466439 2836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:13.466609 2836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:13.468475 2836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "021eb73e52824092a1965bbe3b52e36b"
format_stamp: "Formatted at 2026-08-10 06:20:13 on dist-test-slave-q9h9"
server_key: "00d274966ddb7cf1495f125fd6757532"
server_key_iv: "2874c3e5d6f390f23d6e675621494ebe"
server_key_version: "encryptionkey@0"
I20260810 06:20:13.468997 2836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "021eb73e52824092a1965bbe3b52e36b"
format_stamp: "Formatted at 2026-08-10 06:20:13 on dist-test-slave-q9h9"
server_key: "00d274966ddb7cf1495f125fd6757532"
server_key_iv: "2874c3e5d6f390f23d6e675621494ebe"
server_key_version: "encryptionkey@0"
I20260810 06:20:13.472429 2836 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:20:13.474515 2858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:13.475592 2836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:13.475711 2836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "021eb73e52824092a1965bbe3b52e36b"
format_stamp: "Formatted at 2026-08-10 06:20:13 on dist-test-slave-q9h9"
server_key: "00d274966ddb7cf1495f125fd6757532"
server_key_iv: "2874c3e5d6f390f23d6e675621494ebe"
server_key_version: "encryptionkey@0"
I20260810 06:20:13.475807 2836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:13.500568 2836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:13.503345 2836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:13.503499 2836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:13.512993 2836 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:41717
I20260810 06:20:13.513042 2967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:41717 every 8 connection(s)
I20260810 06:20:13.513832 2836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:20:13.516245 2970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:13.520524 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 2836
I20260810 06:20:13.520645 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:20:13.520917 32563 external_mini_cluster.cc:1428] Setting key 2af85ebc47f156db63753875fc5f5f18
I20260810 06:20:13.521489 2970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b: Bootstrap starting.
I20260810 06:20:13.523733 2970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:13.524411 2970 log.cc:824] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:13.526364 2970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b: No bootstrap required, opened a new log
Aug 10 06:20:13 dist-test-slave-q9h9 krb5kdc[2821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:13.529881 2970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021eb73e52824092a1965bbe3b52e36b" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 41717 } }
I20260810 06:20:13.530151 2970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:13.530215 2970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021eb73e52824092a1965bbe3b52e36b, State: Initialized, Role: FOLLOWER
I20260810 06:20:13.530628 2970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021eb73e52824092a1965bbe3b52e36b" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 41717 } }
I20260810 06:20:13.530753 2970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:20:13.530812 2970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:20:13.530900 2970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:13.531642 2970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021eb73e52824092a1965bbe3b52e36b" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 41717 } }
I20260810 06:20:13.531929 2970 leader_election.cc:302] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 021eb73e52824092a1965bbe3b52e36b; no voters:
I20260810 06:20:13.532191 2970 leader_election.cc:288] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:20:13.532294 2979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:13.532518 2979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b [term 1 LEADER]: Becoming Leader. State: Replica: 021eb73e52824092a1965bbe3b52e36b, State: Running, Role: LEADER
I20260810 06:20:13.532845 2979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021eb73e52824092a1965bbe3b52e36b" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 41717 } }
I20260810 06:20:13.533368 2970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:13.534304 2983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "021eb73e52824092a1965bbe3b52e36b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021eb73e52824092a1965bbe3b52e36b" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 41717 } } }
I20260810 06:20:13.534422 2983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:13.534644 2983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 021eb73e52824092a1965bbe3b52e36b. Latest consensus state: current_term: 1 leader_uuid: "021eb73e52824092a1965bbe3b52e36b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021eb73e52824092a1965bbe3b52e36b" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 41717 } } }
I20260810 06:20:13.534715 2983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:13.534861 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:13.522234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.31.204.254:41717)
0810 06:20:13.522626 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:13.522639 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:13.522671 (+ 32us) server_negotiation.cc:407] Connection header received
0810 06:20:13.523263 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:13.523279 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:13.523579 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:13.523869 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:13.524651 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:13.525942 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:13.526815 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:13.527482 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:13.527661 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:13.529737 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:13.529743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:13.529757 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:13.529785 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:13.531420 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:13.531826 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:13.531832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:13.531838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:13.531944 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:13.532241 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:13.532245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:13.532246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:13.532706 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:13.533458 (+ 752us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:13.533603 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 06:20:13.533883 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":128,"threads_started":1}
I20260810 06:20:13.536036 2994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:13.538483 2994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:13.543715 2994 catalog_manager.cc:1347] Generated new cluster ID: 5846ab0e5abf41ff9ba5bd9535648603
I20260810 06:20:13.543821 2994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:13.554203 2994 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:20:13.554986 2994 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:13.560691 2994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b: Generated new TSK 0
I20260810 06:20:13.561331 2994 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
I20260810 06:20:13.591970 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:41717
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:38195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:13.699798 3012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:13.700001 3012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:13.700045 3012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:13.703891 3012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:13.703936 3012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:13.703981 3012 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 06:20:13.704010 3012 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 06:20:13.704053 3012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:20:13.708961 3012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:38195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.204.254:41717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:13.709807 3012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:13.710693 3012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:13.719091 3020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:13.719097 3019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:13.719105 3024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:13.719385 3012 server_base.cc:1034] running on GCE node
I20260810 06:20:13.719810 3012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:13.720343 3012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:13.721503 3012 hybrid_clock.cc:648] HybridClock initialized: now 1786342813721486 us; error 31 us; skew 500 ppm
Aug 10 06:20:13 dist-test-slave-q9h9 krb5kdc[2821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342813, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:13.724107 3012 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:20:13.725031 3012 webserver.cc:460] Webserver started at http://127.31.204.193:39227/ using document root <none> and password file <none>
I20260810 06:20:13.725594 3012 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:13.725662 3012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:13.725868 3012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:13.727770 3012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "cc6424bc538b4127865238407c5d381c"
format_stamp: "Formatted at 2026-08-10 06:20:13 on dist-test-slave-q9h9"
server_key: "ec152508ce66f046ea3b096d2f41317b"
server_key_iv: "e89f787e9aa5460bc4e352950d8f8e59"
server_key_version: "encryptionkey@0"
I20260810 06:20:13.728274 3012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "cc6424bc538b4127865238407c5d381c"
format_stamp: "Formatted at 2026-08-10 06:20:13 on dist-test-slave-q9h9"
server_key: "ec152508ce66f046ea3b096d2f41317b"
server_key_iv: "e89f787e9aa5460bc4e352950d8f8e59"
server_key_version: "encryptionkey@0"
I20260810 06:20:13.731662 3012 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 06:20:13.733801 3038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:13.734929 3012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:13.735028 3012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "cc6424bc538b4127865238407c5d381c"
format_stamp: "Formatted at 2026-08-10 06:20:13 on dist-test-slave-q9h9"
server_key: "ec152508ce66f046ea3b096d2f41317b"
server_key_iv: "e89f787e9aa5460bc4e352950d8f8e59"
server_key_version: "encryptionkey@0"
I20260810 06:20:13.735118 3012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:13.755895 3012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:13.758282 3012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:13.758432 3012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:13.759701 3012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:13.759750 3012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:13.759800 3012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:13.759845 3012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 06:20:13 dist-test-slave-q9h9 krb5kdc[2821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342813, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:13.769636 3012 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:40473
I20260810 06:20:13.769680 3227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:40473 every 8 connection(s)
I20260810 06:20:13.770510 3012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:20:13.771525 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:13.760308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:56083 (local address 127.31.204.254:41717)
0810 06:20:13.760452 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:13.760458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:13.761583 (+ 1125us) server_negotiation.cc:407] Connection header received
0810 06:20:13.762518 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:13.762521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:13.762575 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:13.762653 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:13.763697 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:13.764207 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:13.765311 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:13.765537 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:13.765638 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:13.768017 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:13.768021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:13.768023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:13.768050 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:13.769530 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:13.770056 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:13.770060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:13.770062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:13.770154 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:13.770456 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:13.770459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:13.770461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:13.770678 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:13.770788 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:13.771289 (+ 501us) server_negotiation.cc:299] Negotiation successful
0810 06:20:13.771407 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 06:20:13.772193 3054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:13.761016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:41717 (local address 127.31.204.193:56083)
0810 06:20:13.761440 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:13.761485 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:13.762320 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:13.762807 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:13.762813 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:13.763107 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:13.763555 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:13.763567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:13.764320 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:13.764324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:13.765190 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:13.765200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:13.765801 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:13.765806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:13.765989 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:13.766507 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:13.766522 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:13.767868 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:13.769640 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:13.769646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:13.769658 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:13.769930 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:13.770245 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:13.770247 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:13.770249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:13.770369 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:13.770769 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:13.770773 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:13.771037 (+ 264us) client_negotiation.cc:769] Sending connection context
0810 06:20:13.771227 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 06:20:13.771447 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
I20260810 06:20:13.773388 3228 heartbeater.cc:347] Connected to a master server at 127.31.204.254:41717
I20260810 06:20:13.773517 3228 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:13.774034 3228 heartbeater.cc:511] Master 127.31.204.254:41717 requested a full tablet report, sending...
I20260810 06:20:13.775520 2891 ts_manager.cc:194] Registered new tserver with Master: cc6424bc538b4127865238407c5d381c (127.31.204.193:40473)
I20260810 06:20:13.776572 2891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:56083
I20260810 06:20:13.777184 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 3012
I20260810 06:20:13.777307 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:20:13.777576 32563 external_mini_cluster.cc:1428] Setting key c63f0f22e44cda6cc0112347056b1b51
I20260810 06:20:13.782174 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:13.775504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:43807 (local address 127.31.204.254:41717)
0810 06:20:13.775612 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:13.775617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:13.775751 (+ 134us) server_negotiation.cc:407] Connection header received
0810 06:20:13.775844 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:13.775847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:13.775883 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:13.775942 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:13.776336 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:13.776802 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:13.777801 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:13.778043 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:13.778140 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:13.779212 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:13.779215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:13.779218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:13.779242 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:13.780770 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:13.781230 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:13.781233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:13.781234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:13.781304 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:13.781539 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:13.781541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:13.781542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:13.781674 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:13.781738 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:13.781966 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 06:20:13.782074 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 06:20:13.782341 3242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:13.775434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:41717 (local address 127.31.204.193:43807)
0810 06:20:13.775680 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:13.775692 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:13.775777 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:13.775966 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:13.775968 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:13.776152 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:13.776237 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:13.776243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:13.776916 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:13.776919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:13.777690 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:13.777701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:13.778171 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:13.778175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:13.778262 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:13.778656 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:13.778669 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:13.779090 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:13.780872 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:13.780874 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:13.780876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:13.781148 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:13.781376 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:13.781378 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:13.781380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:13.781466 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:13.781745 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:13.781747 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:13.781834 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 06:20:13.781934 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 06:20:13.782079 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.31.204.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
I20260810 06:20:13.802276 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:41717
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:38195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:13.905385 3246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:13.905606 3246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:13.905666 3246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:13.909507 3246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:13.909555 3246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:13.909607 3246 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 06:20:13.909636 3246 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 06:20:13.909682 3246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:20:13.914577 3246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:38195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.204.254:41717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:13.915462 3246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:13.916344 3246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:13.924937 3267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:13.924993 3246 server_base.cc:1034] running on GCE node
W20260810 06:20:13.925004 3259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:13.924947 3262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:13.925513 3246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:13.926083 3246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:13.927238 3246 hybrid_clock.cc:648] HybridClock initialized: now 1786342813927222 us; error 30 us; skew 500 ppm
Aug 10 06:20:13 dist-test-slave-q9h9 krb5kdc[2821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342813, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:13.929682 3246 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:20:13.930446 3246 webserver.cc:460] Webserver started at http://127.31.204.194:44385/ using document root <none> and password file <none>
I20260810 06:20:13.930927 3246 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:13.930989 3246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:13.931162 3246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:13.932929 3246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "b036ffb2a7a34ac38869ce9670afeba8"
format_stamp: "Formatted at 2026-08-10 06:20:13 on dist-test-slave-q9h9"
server_key: "49b33c95e259f3ab226176b13d4539b8"
server_key_iv: "9b66d61c610700a341a944265de64f34"
server_key_version: "encryptionkey@0"
I20260810 06:20:13.933422 3246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "b036ffb2a7a34ac38869ce9670afeba8"
format_stamp: "Formatted at 2026-08-10 06:20:13 on dist-test-slave-q9h9"
server_key: "49b33c95e259f3ab226176b13d4539b8"
server_key_iv: "9b66d61c610700a341a944265de64f34"
server_key_version: "encryptionkey@0"
I20260810 06:20:13.936926 3246 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:20:13.939149 3276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:13.940236 3246 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:13.940328 3246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "b036ffb2a7a34ac38869ce9670afeba8"
format_stamp: "Formatted at 2026-08-10 06:20:13 on dist-test-slave-q9h9"
server_key: "49b33c95e259f3ab226176b13d4539b8"
server_key_iv: "9b66d61c610700a341a944265de64f34"
server_key_version: "encryptionkey@0"
I20260810 06:20:13.940423 3246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:13.957213 3246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:13.959649 3246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:13.959802 3246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:13.961067 3246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:13.961122 3246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:13.961159 3246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:13.961189 3246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 06:20:13 dist-test-slave-q9h9 krb5kdc[2821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342813, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:13.970293 3246 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:41177
I20260810 06:20:13.970359 3459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:41177 every 8 connection(s)
I20260810 06:20:13.971074 3246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:20:13.974123 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:13.961802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:57797 (local address 127.31.204.254:41717)
0810 06:20:13.961952 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:13.961957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:13.962717 (+ 760us) server_negotiation.cc:407] Connection header received
0810 06:20:13.963624 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:13.963628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:13.963685 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:13.963740 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:13.965321 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:13.966168 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:13.967845 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:13.968061 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:13.968136 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:13.970823 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:13.970827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:13.970829 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:13.970855 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:13.972217 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:13.972669 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:13.972672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:13.972674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:13.972753 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:13.973085 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:13.973088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:13.973090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:13.973260 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:13.973346 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:13.973820 (+ 474us) server_negotiation.cc:299] Negotiation successful
0810 06:20:13.973946 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 06:20:13.974741 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:13.962124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:41717 (local address 127.31.204.194:57797)
0810 06:20:13.962560 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:13.962607 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:13.963396 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:13.964137 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:13.964146 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:13.964518 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:13.965140 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:13.965160 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:13.966337 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:13.966340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:13.967720 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:13.967731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:13.968212 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:13.968217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:13.968408 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:13.968995 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:13.969015 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:13.970660 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:13.972325 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:13.972331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:13.972344 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:13.972576 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:13.972840 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:13.972843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:13.972845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:13.972999 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:13.973358 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:13.973364 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:13.973584 (+ 220us) client_negotiation.cc:769] Sending connection context
0810 06:20:13.973749 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 06:20:13.974045 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20260810 06:20:13.975750 3460 heartbeater.cc:347] Connected to a master server at 127.31.204.254:41717
I20260810 06:20:13.975884 3460 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:13.976379 3460 heartbeater.cc:511] Master 127.31.204.254:41717 requested a full tablet report, sending...
I20260810 06:20:13.977025 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 3246
I20260810 06:20:13.977149 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:20:13.977469 32563 external_mini_cluster.cc:1428] Setting key 639916bfc873d981084b5c9b176f1392
I20260810 06:20:13.978056 2892 ts_manager.cc:194] Registered new tserver with Master: b036ffb2a7a34ac38869ce9670afeba8 (127.31.204.194:41177)
I20260810 06:20:13.978823 2892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:57797
I20260810 06:20:13.984364 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:13.977831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:44491 (local address 127.31.204.254:41717)
0810 06:20:13.978007 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:13.978011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:13.978258 (+ 247us) server_negotiation.cc:407] Connection header received
0810 06:20:13.978341 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:13.978344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:13.978382 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:13.978460 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:13.978915 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:13.979433 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:13.980276 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:13.980483 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:13.980563 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:13.981644 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:13.981647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:13.981648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:13.981669 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:13.982890 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:13.983322 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:13.983324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:13.983325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:13.983390 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:13.983667 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:13.983670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:13.983672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:13.983864 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:13.983940 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:13.984185 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 06:20:13.984266 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260810 06:20:13.984457 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:13.977726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:41717 (local address 127.31.204.194:44491)
0810 06:20:13.978179 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:13.978192 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:13.978277 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:13.978483 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:13.978486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:13.978668 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:13.978785 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:13.978792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:13.979560 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:13.979562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:13.980162 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:13.980171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:13.980588 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:13.980592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:13.980678 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:13.981111 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:13.981124 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:13.981510 (+ 386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:13.982991 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:13.982994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:13.982996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:13.983222 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:13.983469 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:13.983472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:13.983474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:13.983565 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:13.983950 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:13.983953 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:13.984043 (+ 90us) client_negotiation.cc:769] Sending connection context
0810 06:20:13.984144 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 06:20:13.984276 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.31.204.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
I20260810 06:20:14.002060 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:41717
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:38195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:14.107977 3477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:14.108198 3477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:14.108260 3477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:14.112129 3477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:14.112179 3477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:14.112232 3477 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 06:20:14.112257 3477 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 06:20:14.112303 3477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:20:14.117242 3477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:38195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.204.254:41717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:14.118124 3477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:14.119006 3477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:14.127494 3488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:14.127631 3477 server_base.cc:1034] running on GCE node
W20260810 06:20:14.127503 3487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:14.127851 3490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:14.128211 3477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:14.128756 3477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:14.129913 3477 hybrid_clock.cc:648] HybridClock initialized: now 1786342814129891 us; error 33 us; skew 500 ppm
Aug 10 06:20:14 dist-test-slave-q9h9 krb5kdc[2821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342814, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:14.132378 3477 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:20:14.133159 3477 webserver.cc:460] Webserver started at http://127.31.204.195:36745/ using document root <none> and password file <none>
I20260810 06:20:14.133657 3477 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:14.133731 3477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:14.133939 3477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:14.135802 3477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "e168a43cb2dc432da51e7d1f0fce85f6"
format_stamp: "Formatted at 2026-08-10 06:20:14 on dist-test-slave-q9h9"
server_key: "7ab2cb16a7eb9aad3aadb341e5ccb611"
server_key_iv: "1d0c81f147987e4c20f363fb844d07f4"
server_key_version: "encryptionkey@0"
I20260810 06:20:14.136301 3477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "e168a43cb2dc432da51e7d1f0fce85f6"
format_stamp: "Formatted at 2026-08-10 06:20:14 on dist-test-slave-q9h9"
server_key: "7ab2cb16a7eb9aad3aadb341e5ccb611"
server_key_iv: "1d0c81f147987e4c20f363fb844d07f4"
server_key_version: "encryptionkey@0"
I20260810 06:20:14.139724 3477 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 06:20:14.141861 3502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:14.143030 3477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:14.143129 3477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "e168a43cb2dc432da51e7d1f0fce85f6"
format_stamp: "Formatted at 2026-08-10 06:20:14 on dist-test-slave-q9h9"
server_key: "7ab2cb16a7eb9aad3aadb341e5ccb611"
server_key_iv: "1d0c81f147987e4c20f363fb844d07f4"
server_key_version: "encryptionkey@0"
I20260810 06:20:14.143215 3477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:14.163295 3477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:14.165918 3477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:14.166090 3477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:14.167341 3477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:14.167390 3477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:14.167438 3477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:14.167465 3477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 06:20:14 dist-test-slave-q9h9 krb5kdc[2821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342814, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:14.177639 3477 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:37755
I20260810 06:20:14.177666 3694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:37755 every 8 connection(s)
I20260810 06:20:14.178526 3477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:20:14.180357 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.168089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:50671 (local address 127.31.204.254:41717)
0810 06:20:14.168220 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:14.168224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:14.169141 (+ 917us) server_negotiation.cc:407] Connection header received
0810 06:20:14.169977 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:14.169982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:14.170077 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:14.170170 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:14.171313 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.172099 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.173301 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.173540 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.173630 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.176921 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:14.176943 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:14.176946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:14.176975 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:14.178500 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:14.178983 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:14.178987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:14.178989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:14.179079 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:14.179443 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:14.179446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:14.179447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:14.179649 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:14.179738 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:14.180135 (+ 397us) server_negotiation.cc:299] Negotiation successful
0810 06:20:14.180242 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:20:14.181066 3519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.168567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:41717 (local address 127.31.204.195:50671)
0810 06:20:14.169001 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:14.169039 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:14.169776 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:14.170378 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:14.170387 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:14.170733 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:14.171167 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.171179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.172231 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.172235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.173182 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.173194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.173692 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.173697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.173871 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.174401 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:14.174421 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:14.176777 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:14.178625 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:14.178630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:14.178643 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:14.178883 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:14.179185 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:14.179188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:14.179190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:14.179345 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:14.179760 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:14.179766 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:14.179961 (+ 195us) client_negotiation.cc:769] Sending connection context
0810 06:20:14.180104 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 06:20:14.180316 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":121,"threads_started":1}
I20260810 06:20:14.182092 3695 heartbeater.cc:347] Connected to a master server at 127.31.204.254:41717
I20260810 06:20:14.182232 3695 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:14.182650 3695 heartbeater.cc:511] Master 127.31.204.254:41717 requested a full tablet report, sending...
I20260810 06:20:14.183609 2891 ts_manager.cc:194] Registered new tserver with Master: e168a43cb2dc432da51e7d1f0fce85f6 (127.31.204.195:37755)
I20260810 06:20:14.184304 2891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:50671
I20260810 06:20:14.187047 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 3477
I20260810 06:20:14.187148 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:20:14.187410 32563 external_mini_cluster.cc:1428] Setting key 5098e13c8dc1b0871087996bcfe69c3b
I20260810 06:20:14.189352 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:20:14.190785 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.183904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:41717 (local address 127.31.204.195:40469)
0810 06:20:14.184180 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:14.184204 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:14.184292 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:14.184518 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:14.184521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:14.184685 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:14.184769 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.184774 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.185442 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.185445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.186232 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.186242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.186640 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.186644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.186735 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.187132 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:14.187147 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:14.187453 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:14.189327 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:14.189330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:14.189331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:14.189549 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:14.189875 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:14.189877 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:14.189878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:14.189956 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:14.190406 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:14.190408 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:14.190485 (+ 77us) client_negotiation.cc:769] Sending connection context
0810 06:20:14.190568 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 06:20:14.190664 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":84,"threads_started":1}
I20260810 06:20:14.190887 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.183969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:40469 (local address 127.31.204.254:41717)
0810 06:20:14.184060 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:14.184063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:14.184261 (+ 198us) server_negotiation.cc:407] Connection header received
0810 06:20:14.184365 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:14.184368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:14.184409 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:14.184473 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:14.184874 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.185332 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.186348 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.186545 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.186637 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.187582 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:14.187586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:14.187588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:14.187612 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:14.189103 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:14.189649 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:14.189653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:14.189654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:14.189795 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:14.190065 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:14.190069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:14.190070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:14.190307 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:14.190402 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:14.190636 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 06:20:14.190746 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Aug 10 06:20:14 dist-test-slave-q9h9 krb5kdc[2821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 06:20:14 dist-test-slave-q9h9 krb5kdc[2821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:14.214761 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.207811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43138 (local address 127.31.204.254:41717)
0810 06:20:14.207929 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:14.207933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:14.207943 (+ 10us) server_negotiation.cc:407] Connection header received
0810 06:20:14.207969 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:14.207971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:14.208002 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:14.208063 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:14.208699 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.209166 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.209948 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.210186 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.210267 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.212323 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:14.212326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:14.212328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:14.212349 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:14.213315 (+ 966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:14.213670 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:14.213673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:14.213675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:14.213744 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:14.213984 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:14.213987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:14.213988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:14.214191 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:14.214305 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:14.214518 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 06:20:14.214629 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:20:14.216975 2891 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:20:14.219841 2891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:20:14.233115 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.228134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44654 (local address 127.31.204.195:37755)
0810 06:20:14.228414 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:14.228426 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:14.228447 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:20:14.228515 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:14.228520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:14.228703 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:14.228861 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:14.229246 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.230348 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.231724 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.232477 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.232626 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.232767 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:14.232891 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 06:20:14.232980 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":106,"threads_started":1}
I20260810 06:20:14.233140 3735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.228564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:40473 (local address 127.0.0.1:44112)
0810 06:20:14.229081 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:14.229107 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:14.229190 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:14.229710 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:14.229713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:14.229728 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:14.229822 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.229828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.230997 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.231000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.232014 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.232022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.232814 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.232818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.232907 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.232927 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 06:20:14.232969 (+ 42us) client_negotiation.cc:240] Negotiation successful
0810 06:20:14.233023 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":102,"threads_started":1}
I20260810 06:20:14.233132 3728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.227598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:37755 (local address 127.0.0.1:44654)
0810 06:20:14.228071 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:14.228083 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:14.228152 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:14.229015 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:14.229018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:14.229034 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:14.229109 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.229115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.230464 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.230467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.231599 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.231608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.232599 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.232603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.232712 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.232755 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 06:20:14.232854 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 06:20:14.232959 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":64,"thread_start_us":93,"threads_started":1}
I20260810 06:20:14.233734 3729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.227573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.31.204.194:41177)
0810 06:20:14.228393 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:14.228406 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:14.228429 (+ 23us) server_negotiation.cc:407] Connection header received
0810 06:20:14.228503 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:14.228508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:14.228708 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:14.228860 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:14.229112 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.230300 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.231866 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.232651 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.232775 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.232887 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:14.233549 (+ 662us) server_negotiation.cc:299] Negotiation successful
0810 06:20:14.233629 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":113,"threads_started":1}
I20260810 06:20:14.233913 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.228968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44112 (local address 127.31.204.193:40473)
0810 06:20:14.229388 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:14.229399 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:14.229415 (+ 16us) server_negotiation.cc:407] Connection header received
0810 06:20:14.229471 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:14.229474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:14.229591 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:14.229710 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:14.229915 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.230868 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.232118 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.232683 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.233515 (+ 832us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.233654 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:14.233699 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 06:20:14.233780 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":131,"threads_started":1}
I20260810 06:20:14.234117 3727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.227424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:41177 (local address 127.0.0.1:50752)
0810 06:20:14.227737 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:14.227778 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:14.228032 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:14.228856 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:14.228862 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:14.228884 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:14.228992 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.228999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.230412 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.230415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.231736 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.231747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.232951 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.232955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.233041 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.233098 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 06:20:14.233895 (+ 797us) client_negotiation.cc:240] Negotiation successful
0810 06:20:14.233973 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":72,"threads_started":1}
I20260810 06:20:14.236289 3351 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee88e94c9ae474c9f931c6f86a65bea (DEFAULT_TABLE table=test-table [id=057ca61434ca4ff985c2855bda444f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:14.236186 3588 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee88e94c9ae474c9f931c6f86a65bea (DEFAULT_TABLE table=test-table [id=057ca61434ca4ff985c2855bda444f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:14.236349 3122 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee88e94c9ae474c9f931c6f86a65bea (DEFAULT_TABLE table=test-table [id=057ca61434ca4ff985c2855bda444f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:14.237138 3351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee88e94c9ae474c9f931c6f86a65bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:14.237180 3122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee88e94c9ae474c9f931c6f86a65bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:14.237191 3588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee88e94c9ae474c9f931c6f86a65bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:14.243659 3738 tablet_bootstrap.cc:492] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8: Bootstrap starting.
I20260810 06:20:14.245195 3741 tablet_bootstrap.cc:492] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c: Bootstrap starting.
I20260810 06:20:14.245594 3740 tablet_bootstrap.cc:492] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6: Bootstrap starting.
I20260810 06:20:14.245795 3738 tablet_bootstrap.cc:654] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:14.246723 3738 log.cc:824] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:14.246933 3741 tablet_bootstrap.cc:654] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:14.247557 3740 tablet_bootstrap.cc:654] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:14.247743 3741 log.cc:824] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:14.248346 3740 log.cc:824] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:14.248961 3738 tablet_bootstrap.cc:492] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8: No bootstrap required, opened a new log
I20260810 06:20:14.249158 3738 ts_tablet_manager.cc:1397] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260810 06:20:14.249657 3741 tablet_bootstrap.cc:492] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c: No bootstrap required, opened a new log
I20260810 06:20:14.249825 3741 ts_tablet_manager.cc:1397] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 06:20:14.250197 3740 tablet_bootstrap.cc:492] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6: No bootstrap required, opened a new log
I20260810 06:20:14.250375 3740 ts_tablet_manager.cc:1397] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:20:14.252502 3741 raft_consensus.cc:348] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } } peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } }
I20260810 06:20:14.252710 3741 raft_consensus.cc:374] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:14.252765 3741 raft_consensus.cc:729] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc6424bc538b4127865238407c5d381c, State: Initialized, Role: FOLLOWER
I20260810 06:20:14.252858 3738 raft_consensus.cc:348] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } } peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } }
I20260810 06:20:14.253093 3738 raft_consensus.cc:374] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:14.253140 3738 raft_consensus.cc:729] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b036ffb2a7a34ac38869ce9670afeba8, State: Initialized, Role: FOLLOWER
I20260810 06:20:14.253201 3741 consensus_queue.cc:260] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } } peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } }
I20260810 06:20:14.253539 3738 consensus_queue.cc:260] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } } peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } }
I20260810 06:20:14.253840 3228 heartbeater.cc:503] Master 127.31.204.254:41717 was elected leader, sending a full tablet report...
I20260810 06:20:14.254173 3741 ts_tablet_manager.cc:1428] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260810 06:20:14.253980 3740 raft_consensus.cc:348] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } } peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } }
I20260810 06:20:14.254243 3740 raft_consensus.cc:374] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:14.254293 3740 raft_consensus.cc:729] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e168a43cb2dc432da51e7d1f0fce85f6, State: Initialized, Role: FOLLOWER
I20260810 06:20:14.254483 3738 ts_tablet_manager.cc:1428] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8: Time spent starting tablet: real 0.005s user 0.005s sys 0.002s
I20260810 06:20:14.254674 3740 consensus_queue.cc:260] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } } peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } }
I20260810 06:20:14.254820 3460 heartbeater.cc:503] Master 127.31.204.254:41717 was elected leader, sending a full tablet report...
I20260810 06:20:14.255586 3740 ts_tablet_manager.cc:1428] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 06:20:14.255889 3695 heartbeater.cc:503] Master 127.31.204.254:41717 was elected leader, sending a full tablet report...
W20260810 06:20:14.271592 3231 tablet.cc:2369] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:20:14.429716 3697 tablet.cc:2369] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:20:14.472136 3465 tablet.cc:2369] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:20:14.489732 3747 raft_consensus.cc:482] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:20:14.489923 3747 raft_consensus.cc:504] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } } peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } }
I20260810 06:20:14.491060 3747 leader_election.cc:288] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b036ffb2a7a34ac38869ce9670afeba8 (127.31.204.194:41177), e168a43cb2dc432da51e7d1f0fce85f6 (127.31.204.195:37755)
I20260810 06:20:14.494768 3729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.491396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:39665 (local address 127.31.204.194:41177)
0810 06:20:14.491650 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:14.491655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:14.491671 (+ 16us) server_negotiation.cc:407] Connection header received
0810 06:20:14.491800 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:14.491803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:14.491850 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:14.491915 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:14.492191 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.492702 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.493842 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.494267 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.494362 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.494416 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:14.494576 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 06:20:14.494642 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260810 06:20:14.494792 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.491418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:45789 (local address 127.31.204.195:37755)
0810 06:20:14.491553 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:14.491557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:14.491796 (+ 239us) server_negotiation.cc:407] Connection header received
0810 06:20:14.491909 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:14.491912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:14.491963 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:14.492038 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:14.492263 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.492751 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.493824 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.494288 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.494401 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.494452 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:14.494613 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 06:20:14.494685 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 06:20:14.494831 3753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.491319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:37755 (local address 127.31.204.193:45789)
0810 06:20:14.491701 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:14.491714 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:14.491823 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:14.492054 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:14.492056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:14.492071 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:14.492154 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.492159 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.492857 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.492859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.493708 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.493717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.494405 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.494408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.494502 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.494519 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 06:20:14.494590 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 06:20:14.494651 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":59,"thread_start_us":74,"threads_started":1}
I20260810 06:20:14.494831 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.491318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:41177 (local address 127.31.204.193:39665)
0810 06:20:14.491587 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:14.491603 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:14.491689 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:14.491945 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:14.491947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:14.491964 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:14.492092 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.492098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.492803 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.492806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.493726 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.493736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.494371 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.494375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.494461 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.494488 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 06:20:14.494547 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 06:20:14.494611 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":92,"thread_start_us":103,"threads_started":1}
I20260810 06:20:14.495510 3380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee88e94c9ae474c9f931c6f86a65bea" candidate_uuid: "cc6424bc538b4127865238407c5d381c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b036ffb2a7a34ac38869ce9670afeba8" is_pre_election: true
I20260810 06:20:14.495590 3613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee88e94c9ae474c9f931c6f86a65bea" candidate_uuid: "cc6424bc538b4127865238407c5d381c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" is_pre_election: true
I20260810 06:20:14.495820 3380 raft_consensus.cc:2454] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc6424bc538b4127865238407c5d381c in term 0.
I20260810 06:20:14.495820 3613 raft_consensus.cc:2454] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc6424bc538b4127865238407c5d381c in term 0.
I20260810 06:20:14.496284 3045 leader_election.cc:302] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc6424bc538b4127865238407c5d381c, e168a43cb2dc432da51e7d1f0fce85f6; no voters:
I20260810 06:20:14.496533 3747 raft_consensus.cc:2781] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:20:14.496609 3747 raft_consensus.cc:482] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:20:14.496639 3747 raft_consensus.cc:3037] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:14.497625 3747 raft_consensus.cc:504] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } } peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } }
I20260810 06:20:14.497943 3747 leader_election.cc:288] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [CANDIDATE]: Term 1 election: Requested vote from peers b036ffb2a7a34ac38869ce9670afeba8 (127.31.204.194:41177), e168a43cb2dc432da51e7d1f0fce85f6 (127.31.204.195:37755)
I20260810 06:20:14.498271 3380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee88e94c9ae474c9f931c6f86a65bea" candidate_uuid: "cc6424bc538b4127865238407c5d381c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b036ffb2a7a34ac38869ce9670afeba8"
I20260810 06:20:14.498319 3613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee88e94c9ae474c9f931c6f86a65bea" candidate_uuid: "cc6424bc538b4127865238407c5d381c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e168a43cb2dc432da51e7d1f0fce85f6"
I20260810 06:20:14.498411 3613 raft_consensus.cc:3037] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:14.498404 3380 raft_consensus.cc:3037] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:14.499662 3380 raft_consensus.cc:2454] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc6424bc538b4127865238407c5d381c in term 1.
I20260810 06:20:14.499985 3613 raft_consensus.cc:2454] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc6424bc538b4127865238407c5d381c in term 1.
I20260810 06:20:14.499981 3047 leader_election.cc:302] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b036ffb2a7a34ac38869ce9670afeba8, cc6424bc538b4127865238407c5d381c; no voters:
I20260810 06:20:14.500144 3747 raft_consensus.cc:2781] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:14.500335 3747 raft_consensus.cc:686] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [term 1 LEADER]: Becoming Leader. State: Replica: cc6424bc538b4127865238407c5d381c, State: Running, Role: LEADER
I20260810 06:20:14.500646 3747 consensus_queue.cc:237] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } } peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } }
I20260810 06:20:14.503537 2890 catalog_manager.cc:5495] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c reported cstate change: term changed from 0 to 1, leader changed from <none> to cc6424bc538b4127865238407c5d381c (127.31.204.193). New cstate: current_term: 1 leader_uuid: "cc6424bc538b4127865238407c5d381c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } health_report { overall_health: HEALTHY } } }
Aug 10 06:20:14 dist-test-slave-q9h9 krb5kdc[2821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342813, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:14.538921 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.531354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:41717 (local address 127.0.0.1:43154)
0810 06:20:14.531609 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:14.531620 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:14.531742 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:14.531945 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:14.531947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:14.532311 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:14.532397 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.532403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.533095 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.533098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.533713 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.533724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.534127 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.534131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.534208 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.534772 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:14.534784 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:14.536142 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:14.537426 (+ 1284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:14.537435 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:14.537441 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:14.537660 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:14.537907 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:14.537910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:14.537912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:14.537986 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:14.538390 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:14.538396 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:14.538498 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 06:20:14.538589 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 06:20:14.538744 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":88,"threads_started":1}
I20260810 06:20:14.538921 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.531407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43154 (local address 127.31.204.254:41717)
0810 06:20:14.531510 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:14.531514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:14.531700 (+ 186us) server_negotiation.cc:407] Connection header received
0810 06:20:14.531817 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:14.531820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:14.531857 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:14.531917 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:14.532502 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.533001 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.533811 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.534035 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.534127 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.536265 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:14.536268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:14.536271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:14.536294 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:14.537333 (+ 1039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:14.537754 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:14.537757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:14.537759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:14.537829 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:14.538081 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:14.538085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:14.538087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:14.538294 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:14.538378 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:14.538647 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 06:20:14.538801 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":33}
I20260810 06:20:14.542378 2890 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:43154:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 06:20:14.542981 2890 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:20:14.547231 3122 tablet_service.cc:1467] Processing CreateTablet for tablet 79e38bd21d4c42b9ac8b333bb441e325 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9c1bfe7eb9a048eab4406417f7135141]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 06:20:14.547364 3351 tablet_service.cc:1467] Processing CreateTablet for tablet 79e38bd21d4c42b9ac8b333bb441e325 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9c1bfe7eb9a048eab4406417f7135141]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 06:20:14.547868 3122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79e38bd21d4c42b9ac8b333bb441e325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:14.547631 3588 tablet_service.cc:1467] Processing CreateTablet for tablet 79e38bd21d4c42b9ac8b333bb441e325 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9c1bfe7eb9a048eab4406417f7135141]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 06:20:14.548000 3351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79e38bd21d4c42b9ac8b333bb441e325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:14.548235 3588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79e38bd21d4c42b9ac8b333bb441e325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:14.550225 3741 tablet_bootstrap.cc:492] T 79e38bd21d4c42b9ac8b333bb441e325 P cc6424bc538b4127865238407c5d381c: Bootstrap starting.
I20260810 06:20:14.550839 3738 tablet_bootstrap.cc:492] T 79e38bd21d4c42b9ac8b333bb441e325 P b036ffb2a7a34ac38869ce9670afeba8: Bootstrap starting.
I20260810 06:20:14.551151 3740 tablet_bootstrap.cc:492] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6: Bootstrap starting.
I20260810 06:20:14.551527 3741 tablet_bootstrap.cc:654] T 79e38bd21d4c42b9ac8b333bb441e325 P cc6424bc538b4127865238407c5d381c: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:14.551739 3738 tablet_bootstrap.cc:654] T 79e38bd21d4c42b9ac8b333bb441e325 P b036ffb2a7a34ac38869ce9670afeba8: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:14.551887 3740 tablet_bootstrap.cc:654] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:14.552819 3741 tablet_bootstrap.cc:492] T 79e38bd21d4c42b9ac8b333bb441e325 P cc6424bc538b4127865238407c5d381c: No bootstrap required, opened a new log
I20260810 06:20:14.552903 3741 ts_tablet_manager.cc:1397] T 79e38bd21d4c42b9ac8b333bb441e325 P cc6424bc538b4127865238407c5d381c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 06:20:14.553074 3740 tablet_bootstrap.cc:492] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6: No bootstrap required, opened a new log
I20260810 06:20:14.553174 3740 ts_tablet_manager.cc:1397] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:14.553298 3741 raft_consensus.cc:348] T 79e38bd21d4c42b9ac8b333bb441e325 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:14.553386 3741 raft_consensus.cc:374] T 79e38bd21d4c42b9ac8b333bb441e325 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:14.553416 3741 raft_consensus.cc:729] T 79e38bd21d4c42b9ac8b333bb441e325 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc6424bc538b4127865238407c5d381c, State: Initialized, Role: FOLLOWER
I20260810 06:20:14.553534 3741 consensus_queue.cc:260] T 79e38bd21d4c42b9ac8b333bb441e325 P cc6424bc538b4127865238407c5d381c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:14.553570 3740 raft_consensus.cc:348] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:14.553663 3740 raft_consensus.cc:374] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:14.553697 3740 raft_consensus.cc:729] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e168a43cb2dc432da51e7d1f0fce85f6, State: Initialized, Role: FOLLOWER
I20260810 06:20:14.553812 3741 ts_tablet_manager.cc:1428] T 79e38bd21d4c42b9ac8b333bb441e325 P cc6424bc538b4127865238407c5d381c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:14.553820 3740 consensus_queue.cc:260] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:14.554075 3740 ts_tablet_manager.cc:1428] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:14.554068 3747 tablet_replica.cc:439] T 79e38bd21d4c42b9ac8b333bb441e325 P cc6424bc538b4127865238407c5d381c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } } }
I20260810 06:20:14.554083 3751 tablet_replica.cc:439] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } } }
I20260810 06:20:14.554190 3747 tablet_replica.cc:442] T 79e38bd21d4c42b9ac8b333bb441e325 P cc6424bc538b4127865238407c5d381c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:14.554224 3751 tablet_replica.cc:442] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:14.555054 3738 tablet_bootstrap.cc:492] T 79e38bd21d4c42b9ac8b333bb441e325 P b036ffb2a7a34ac38869ce9670afeba8: No bootstrap required, opened a new log
I20260810 06:20:14.555148 3738 ts_tablet_manager.cc:1397] T 79e38bd21d4c42b9ac8b333bb441e325 P b036ffb2a7a34ac38869ce9670afeba8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 06:20:14.555598 3738 raft_consensus.cc:348] T 79e38bd21d4c42b9ac8b333bb441e325 P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:14.555709 3738 raft_consensus.cc:374] T 79e38bd21d4c42b9ac8b333bb441e325 P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:14.555740 3738 raft_consensus.cc:729] T 79e38bd21d4c42b9ac8b333bb441e325 P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b036ffb2a7a34ac38869ce9670afeba8, State: Initialized, Role: FOLLOWER
I20260810 06:20:14.555857 3738 consensus_queue.cc:260] T 79e38bd21d4c42b9ac8b333bb441e325 P b036ffb2a7a34ac38869ce9670afeba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:14.556066 3738 ts_tablet_manager.cc:1428] T 79e38bd21d4c42b9ac8b333bb441e325 P b036ffb2a7a34ac38869ce9670afeba8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:14.556085 3749 tablet_replica.cc:439] T 79e38bd21d4c42b9ac8b333bb441e325 P b036ffb2a7a34ac38869ce9670afeba8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } } }
I20260810 06:20:14.556221 3749 tablet_replica.cc:442] T 79e38bd21d4c42b9ac8b333bb441e325 P b036ffb2a7a34ac38869ce9670afeba8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:14.573881 3751 raft_consensus.cc:482] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:20:14.573966 3751 raft_consensus.cc:504] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:14.574928 3751 leader_election.cc:288] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc6424bc538b4127865238407c5d381c (127.31.204.193:40473), b036ffb2a7a34ac38869ce9670afeba8 (127.31.204.194:41177)
I20260810 06:20:14.578452 3729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.575209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:33479 (local address 127.31.204.194:41177)
0810 06:20:14.575366 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:14.575371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:14.575388 (+ 17us) server_negotiation.cc:407] Connection header received
0810 06:20:14.575432 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:14.575435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:14.575478 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:14.575549 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:14.575873 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.576494 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.577542 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.577927 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.578064 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.578113 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:14.578267 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 06:20:14.578334 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 06:20:14.578456 3519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.575147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:41177 (local address 127.31.204.195:33479)
0810 06:20:14.575257 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:14.575271 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:14.575348 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:14.575575 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:14.575578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:14.575614 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:14.575747 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.575754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.576591 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.576594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.577429 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.577438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.578053 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.578056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.578158 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.578175 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 06:20:14.578243 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 06:20:14.578342 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":22,"spinlock_wait_cycles":33280}
I20260810 06:20:14.578918 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.575271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:36517 (local address 127.31.204.193:40473)
0810 06:20:14.575373 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:14.575377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:14.575587 (+ 210us) server_negotiation.cc:407] Connection header received
0810 06:20:14.575698 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:14.575701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:14.575745 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:14.575816 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:14.576071 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.576649 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.577701 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.578154 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.578277 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.578342 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:14.578745 (+ 403us) server_negotiation.cc:299] Negotiation successful
0810 06:20:14.578817 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:20:14.578930 3774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.575151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:40473 (local address 127.31.204.195:36517)
0810 06:20:14.575504 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:14.575513 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:14.575618 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:14.575824 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:14.575827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:14.575844 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:14.575953 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.575981 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.576758 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.576762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.577579 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.577588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.578439 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.578441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.578515 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.578653 (+ 138us) client_negotiation.cc:769] Sending connection context
0810 06:20:14.578723 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 06:20:14.578788 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":125,"spinlock_wait_cycles":3328,"thread_start_us":98,"threads_started":1}
I20260810 06:20:14.578895 3380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e38bd21d4c42b9ac8b333bb441e325" candidate_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b036ffb2a7a34ac38869ce9670afeba8" is_pre_election: true
I20260810 06:20:14.579008 3380 raft_consensus.cc:2454] T 79e38bd21d4c42b9ac8b333bb441e325 P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e168a43cb2dc432da51e7d1f0fce85f6 in term 0.
I20260810 06:20:14.579295 3513 leader_election.cc:302] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b036ffb2a7a34ac38869ce9670afeba8, e168a43cb2dc432da51e7d1f0fce85f6; no voters:
I20260810 06:20:14.579433 3152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e38bd21d4c42b9ac8b333bb441e325" candidate_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc6424bc538b4127865238407c5d381c" is_pre_election: true
I20260810 06:20:14.579535 3751 raft_consensus.cc:2781] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:20:14.579600 3751 raft_consensus.cc:482] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:20:14.579644 3751 raft_consensus.cc:3037] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:14.579619 3152 raft_consensus.cc:2454] T 79e38bd21d4c42b9ac8b333bb441e325 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e168a43cb2dc432da51e7d1f0fce85f6 in term 0.
I20260810 06:20:14.580565 3751 raft_consensus.cc:504] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:14.580871 3751 leader_election.cc:288] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 [CANDIDATE]: Term 1 election: Requested vote from peers cc6424bc538b4127865238407c5d381c (127.31.204.193:40473), b036ffb2a7a34ac38869ce9670afeba8 (127.31.204.194:41177)
I20260810 06:20:14.581149 3152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e38bd21d4c42b9ac8b333bb441e325" candidate_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc6424bc538b4127865238407c5d381c"
I20260810 06:20:14.581199 3380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e38bd21d4c42b9ac8b333bb441e325" candidate_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b036ffb2a7a34ac38869ce9670afeba8"
I20260810 06:20:14.581246 3152 raft_consensus.cc:3037] T 79e38bd21d4c42b9ac8b333bb441e325 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:14.581310 3380 raft_consensus.cc:3037] T 79e38bd21d4c42b9ac8b333bb441e325 P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:14.582171 3380 raft_consensus.cc:2454] T 79e38bd21d4c42b9ac8b333bb441e325 P b036ffb2a7a34ac38869ce9670afeba8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e168a43cb2dc432da51e7d1f0fce85f6 in term 1.
I20260810 06:20:14.582204 3152 raft_consensus.cc:2454] T 79e38bd21d4c42b9ac8b333bb441e325 P cc6424bc538b4127865238407c5d381c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e168a43cb2dc432da51e7d1f0fce85f6 in term 1.
I20260810 06:20:14.582415 3513 leader_election.cc:302] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b036ffb2a7a34ac38869ce9670afeba8, cc6424bc538b4127865238407c5d381c, e168a43cb2dc432da51e7d1f0fce85f6; no voters:
I20260810 06:20:14.582595 3751 raft_consensus.cc:2781] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:14.582836 3751 raft_consensus.cc:686] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 LEADER]: Becoming Leader. State: Replica: e168a43cb2dc432da51e7d1f0fce85f6, State: Running, Role: LEADER
I20260810 06:20:14.583129 3751 consensus_queue.cc:237] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:14.585129 3775 tablet_replica.cc:439] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6: TxnStatusTablet state changed. Reason: New leader e168a43cb2dc432da51e7d1f0fce85f6. Latest consensus state: current_term: 1 leader_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } } }
I20260810 06:20:14.585233 3775 tablet_replica.cc:442] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6: This TxnStatusTablet replica's current role is: LEADER
I20260810 06:20:14.585494 3777 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 06:20:14.585690 3777 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 06:20:14.586198 2890 catalog_manager.cc:5495] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e168a43cb2dc432da51e7d1f0fce85f6 (127.31.204.195). New cstate: current_term: 1 leader_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } health_report { overall_health: HEALTHY } } }
I20260810 06:20:14.599860 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.596594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:37755 (local address 127.0.0.1:44662)
0810 06:20:14.596739 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:14.596753 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:14.596851 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:14.597111 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:14.597114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:14.597122 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:20:14.597210 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.597215 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.597866 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.597869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.598496 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.598505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.598917 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.598921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.599003 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.599044 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:20:14.599596 (+ 552us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:20:14.599612 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 06:20:14.599672 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 06:20:14.599753 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 06:20:14.599856 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.596666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.31.204.195:37755)
0810 06:20:14.596799 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:14.596803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:14.596815 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:20:14.596968 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:14.596970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:14.597015 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:14.597090 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:14.597301 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.597769 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.598599 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.598819 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.598903 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.599166 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:14.599502 (+ 336us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:14.599596 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:14.599689 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 06:20:14.599731 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 06:20:14.605082 3152 raft_consensus.cc:1261] T 79e38bd21d4c42b9ac8b333bb441e325 P cc6424bc538b4127865238407c5d381c [term 1 FOLLOWER]: Refusing update from remote peer e168a43cb2dc432da51e7d1f0fce85f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:14.605119 3380 raft_consensus.cc:1261] T 79e38bd21d4c42b9ac8b333bb441e325 P b036ffb2a7a34ac38869ce9670afeba8 [term 1 FOLLOWER]: Refusing update from remote peer e168a43cb2dc432da51e7d1f0fce85f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:14.605715 3775 consensus_queue.cc:1035] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:14.605903 3751 consensus_queue.cc:1035] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:14.609690 3747 tablet_replica.cc:439] T 79e38bd21d4c42b9ac8b333bb441e325 P cc6424bc538b4127865238407c5d381c: TxnStatusTablet state changed. Reason: New leader e168a43cb2dc432da51e7d1f0fce85f6. Latest consensus state: current_term: 1 leader_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } } }
I20260810 06:20:14.609809 3747 tablet_replica.cc:442] T 79e38bd21d4c42b9ac8b333bb441e325 P cc6424bc538b4127865238407c5d381c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:14.611691 3749 tablet_replica.cc:439] T 79e38bd21d4c42b9ac8b333bb441e325 P b036ffb2a7a34ac38869ce9670afeba8: TxnStatusTablet state changed. Reason: New leader e168a43cb2dc432da51e7d1f0fce85f6. Latest consensus state: current_term: 1 leader_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } } }
I20260810 06:20:14.611819 3749 tablet_replica.cc:442] T 79e38bd21d4c42b9ac8b333bb441e325 P b036ffb2a7a34ac38869ce9670afeba8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:14.613780 3751 tablet_replica.cc:439] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } } }
I20260810 06:20:14.613881 3751 tablet_replica.cc:442] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6: This TxnStatusTablet replica's current role is: LEADER
I20260810 06:20:14.614117 3747 tablet_replica.cc:439] T 79e38bd21d4c42b9ac8b333bb441e325 P cc6424bc538b4127865238407c5d381c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } } }
I20260810 06:20:14.614224 3747 tablet_replica.cc:442] T 79e38bd21d4c42b9ac8b333bb441e325 P cc6424bc538b4127865238407c5d381c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:14.614377 3775 tablet_replica.cc:439] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } } }
I20260810 06:20:14.614465 3775 tablet_replica.cc:442] T 79e38bd21d4c42b9ac8b333bb441e325 P e168a43cb2dc432da51e7d1f0fce85f6: This TxnStatusTablet replica's current role is: LEADER
I20260810 06:20:14.615103 3785 mvcc.cc:205] Tried to move back new op lower bound from 7316860168615866368 to 7316860168534761472. Current Snapshot: MvccSnapshot[applied={T|T < 7316860168615866368}]
I20260810 06:20:14.615654 3749 tablet_replica.cc:439] T 79e38bd21d4c42b9ac8b333bb441e325 P b036ffb2a7a34ac38869ce9670afeba8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } } }
I20260810 06:20:14.615756 3749 tablet_replica.cc:442] T 79e38bd21d4c42b9ac8b333bb441e325 P b036ffb2a7a34ac38869ce9670afeba8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:14.618834 3783 mvcc.cc:205] Tried to move back new op lower bound from 7316860168615866368 to 7316860168534761472. Current Snapshot: MvccSnapshot[applied={T|T < 7316860168615866368 or (T in {7316860168615866368})}]
I20260810 06:20:14.619052 3792 mvcc.cc:205] Tried to move back new op lower bound from 7316860168615866368 to 7316860168534761472. Current Snapshot: MvccSnapshot[applied={T|T < 7316860168615866368 or (T in {7316860168615866368})}]
I20260810 06:20:14.624233 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.620939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44118 (local address 127.31.204.193:40473)
0810 06:20:14.621038 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:14.621042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:14.621076 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:20:14.621242 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:14.621246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:14.621299 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:14.621379 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:14.621617 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.622141 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.622962 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.623186 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.623271 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.623529 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:14.623892 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:14.623986 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:14.624066 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 06:20:14.624115 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 06:20:14.634480 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.631022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:49209 (local address 127.31.204.195:37755)
0810 06:20:14.631144 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:14.631148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:14.631333 (+ 185us) server_negotiation.cc:407] Connection header received
0810 06:20:14.631439 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:14.631442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:14.631481 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:14.631536 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:14.631768 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.632222 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.633456 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.633662 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.633775 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.634104 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:14.634163 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:14.634215 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:14.634316 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 06:20:14.634356 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 06:20:14.634543 3809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.630941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:37755 (local address 127.31.204.193:49209)
0810 06:20:14.631257 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:14.631270 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:14.631368 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:14.631560 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:14.631563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:14.631570 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:20:14.631686 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.631692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.632312 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.632314 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.633307 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.633318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.633778 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.633782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.633876 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.633926 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:20:14.634234 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:20:14.634239 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 06:20:14.634310 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 06:20:14.634383 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
I20260810 06:20:14.641326 3613 raft_consensus.cc:1261] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 FOLLOWER]: Refusing update from remote peer cc6424bc538b4127865238407c5d381c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:14.641757 3380 raft_consensus.cc:1261] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8 [term 1 FOLLOWER]: Refusing update from remote peer cc6424bc538b4127865238407c5d381c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:14.641872 3757 consensus_queue.cc:1035] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:14.642236 3757 consensus_queue.cc:1035] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:14.664318 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.660933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:54197 (local address 127.31.204.193:40473)
0810 06:20:14.661216 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:14.661219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:14.661229 (+ 10us) server_negotiation.cc:407] Connection header received
0810 06:20:14.661258 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:14.661261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:14.661296 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:14.661351 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:14.661581 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.662247 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.663437 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.663636 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.663708 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.663986 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:14.664046 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:14.664092 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:14.664184 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 06:20:14.664222 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221}
I20260810 06:20:14.664388 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.660869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:40473 (local address 127.31.204.195:54197)
0810 06:20:14.661015 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:14.661029 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:14.661097 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:14.661380 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:14.661383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:14.661390 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:20:14.661490 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.661495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.662374 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.662377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.663326 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.663336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.663731 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.663735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.663830 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.663880 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:20:14.664113 (+ 233us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:20:14.664119 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 06:20:14.664179 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 06:20:14.664262 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 06:20:14.684861 2890 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:14.685020 2890 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:14.687143 2890 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b: Sending DeleteTablet for 3 replicas of tablet 9ee88e94c9ae474c9f931c6f86a65bea
I20260810 06:20:14.687718 3351 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ee88e94c9ae474c9f931c6f86a65bea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:14 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:50752
I20260810 06:20:14.688196 3824 tablet_replica.cc:330] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8: stopping tablet replica
I20260810 06:20:14.688385 3122 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ee88e94c9ae474c9f931c6f86a65bea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:14 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:44112
I20260810 06:20:14.688496 3824 raft_consensus.cc:2229] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:14.688680 3824 raft_consensus.cc:2258] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:14.688896 3588 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ee88e94c9ae474c9f931c6f86a65bea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:14 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:44654
I20260810 06:20:14.689985 2890 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 06:20:14.690105 3824 ts_tablet_manager.cc:1905] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 06:20:14.690385 2890 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:20:14.692135 3825 tablet_replica.cc:330] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c: stopping tablet replica
I20260810 06:20:14.692370 3826 tablet_replica.cc:330] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6: stopping tablet replica
I20260810 06:20:14.692705 3826 raft_consensus.cc:2229] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:14.693039 3825 raft_consensus.cc:2229] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:20:14.693257 3826 raft_consensus.cc:2258] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:14.693625 3825 raft_consensus.cc:2258] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:14.694882 3824 ts_tablet_manager.cc:1918] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:20:14.695705 3824 log.cc:1196] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/9ee88e94c9ae474c9f931c6f86a65bea
I20260810 06:20:14.696039 3824 ts_tablet_manager.cc:1939] T 9ee88e94c9ae474c9f931c6f86a65bea P b036ffb2a7a34ac38869ce9670afeba8: Deleting consensus metadata
I20260810 06:20:14.696468 3122 tablet_service.cc:1467] Processing CreateTablet for tablet 86eb1274ac8d47d5b83d63953a3be22e (DEFAULT_TABLE table=test-table [id=f2e8591512a640cfbe35f6e0844453aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:14.696710 3122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86eb1274ac8d47d5b83d63953a3be22e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:14.698318 3351 tablet_service.cc:1467] Processing CreateTablet for tablet 86eb1274ac8d47d5b83d63953a3be22e (DEFAULT_TABLE table=test-table [id=f2e8591512a640cfbe35f6e0844453aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:14.698531 3351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86eb1274ac8d47d5b83d63953a3be22e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:14.699083 3741 tablet_bootstrap.cc:492] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c: Bootstrap starting.
I20260810 06:20:14.700006 3741 tablet_bootstrap.cc:654] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:14.700596 3738 tablet_bootstrap.cc:492] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8: Bootstrap starting.
I20260810 06:20:14.700754 3588 tablet_service.cc:1467] Processing CreateTablet for tablet 86eb1274ac8d47d5b83d63953a3be22e (DEFAULT_TABLE table=test-table [id=f2e8591512a640cfbe35f6e0844453aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:14.700995 3588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86eb1274ac8d47d5b83d63953a3be22e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:14.701435 3741 tablet_bootstrap.cc:492] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c: No bootstrap required, opened a new log
I20260810 06:20:14.701471 3738 tablet_bootstrap.cc:654] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:14.701504 3741 ts_tablet_manager.cc:1397] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:14.701885 3741 raft_consensus.cc:348] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:14.701982 3741 raft_consensus.cc:374] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:14.702033 3741 raft_consensus.cc:729] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc6424bc538b4127865238407c5d381c, State: Initialized, Role: FOLLOWER
I20260810 06:20:14.702147 3741 consensus_queue.cc:260] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:14.702404 3741 ts_tablet_manager.cc:1428] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:14.703313 3740 tablet_bootstrap.cc:492] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6: Bootstrap starting.
I20260810 06:20:14.703919 2865 catalog_manager.cc:4841] TS b036ffb2a7a34ac38869ce9670afeba8 (127.31.204.194:41177): tablet 9ee88e94c9ae474c9f931c6f86a65bea (table test-table [id=057ca61434ca4ff985c2855bda444f53]) successfully deleted
I20260810 06:20:14.704114 3740 tablet_bootstrap.cc:654] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:14.705678 3738 tablet_bootstrap.cc:492] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8: No bootstrap required, opened a new log
I20260810 06:20:14.705775 3738 ts_tablet_manager.cc:1397] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260810 06:20:14.705919 3825 ts_tablet_manager.cc:1905] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:14.706200 3738 raft_consensus.cc:348] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:14.706310 3738 raft_consensus.cc:374] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:14.706344 3738 raft_consensus.cc:729] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b036ffb2a7a34ac38869ce9670afeba8, State: Initialized, Role: FOLLOWER
I20260810 06:20:14.706452 3738 consensus_queue.cc:260] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:14.706485 3826 ts_tablet_manager.cc:1905] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:14.706702 3738 ts_tablet_manager.cc:1428] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:14.709863 3825 ts_tablet_manager.cc:1918] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:20:14.709947 3825 log.cc:1196] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/wal/wals/9ee88e94c9ae474c9f931c6f86a65bea
I20260810 06:20:14.710299 3825 ts_tablet_manager.cc:1939] T 9ee88e94c9ae474c9f931c6f86a65bea P cc6424bc538b4127865238407c5d381c: Deleting consensus metadata
I20260810 06:20:14.711395 2863 catalog_manager.cc:4841] TS cc6424bc538b4127865238407c5d381c (127.31.204.193:40473): tablet 9ee88e94c9ae474c9f931c6f86a65bea (table test-table [id=057ca61434ca4ff985c2855bda444f53]) successfully deleted
I20260810 06:20:14.711807 3826 ts_tablet_manager.cc:1918] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:20:14.711889 3826 log.cc:1196] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/9ee88e94c9ae474c9f931c6f86a65bea
I20260810 06:20:14.712214 3826 ts_tablet_manager.cc:1939] T 9ee88e94c9ae474c9f931c6f86a65bea P e168a43cb2dc432da51e7d1f0fce85f6: Deleting consensus metadata
I20260810 06:20:14.713200 2863 catalog_manager.cc:4841] TS e168a43cb2dc432da51e7d1f0fce85f6 (127.31.204.195:37755): tablet 9ee88e94c9ae474c9f931c6f86a65bea (table test-table [id=057ca61434ca4ff985c2855bda444f53]) successfully deleted
I20260810 06:20:14.714633 3740 tablet_bootstrap.cc:492] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6: No bootstrap required, opened a new log
I20260810 06:20:14.714732 3740 ts_tablet_manager.cc:1397] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260810 06:20:14.715097 3740 raft_consensus.cc:348] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:14.715186 3740 raft_consensus.cc:374] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:14.715216 3740 raft_consensus.cc:729] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e168a43cb2dc432da51e7d1f0fce85f6, State: Initialized, Role: FOLLOWER
I20260810 06:20:14.715342 3740 consensus_queue.cc:260] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:14.715607 3740 ts_tablet_manager.cc:1428] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 06:20:14.822788 3819 raft_consensus.cc:482] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:20:14.822923 3819 raft_consensus.cc:504] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:14.823354 3819 leader_election.cc:288] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc6424bc538b4127865238407c5d381c (127.31.204.193:40473), b036ffb2a7a34ac38869ce9670afeba8 (127.31.204.194:41177)
I20260810 06:20:14.823796 3380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86eb1274ac8d47d5b83d63953a3be22e" candidate_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b036ffb2a7a34ac38869ce9670afeba8" is_pre_election: true
I20260810 06:20:14.823792 3152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86eb1274ac8d47d5b83d63953a3be22e" candidate_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc6424bc538b4127865238407c5d381c" is_pre_election: true
I20260810 06:20:14.823935 3380 raft_consensus.cc:2454] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e168a43cb2dc432da51e7d1f0fce85f6 in term 0.
I20260810 06:20:14.823971 3152 raft_consensus.cc:2454] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e168a43cb2dc432da51e7d1f0fce85f6 in term 0.
I20260810 06:20:14.824278 3509 leader_election.cc:302] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc6424bc538b4127865238407c5d381c, e168a43cb2dc432da51e7d1f0fce85f6; no voters:
I20260810 06:20:14.824466 3819 raft_consensus.cc:2781] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:20:14.824518 3819 raft_consensus.cc:482] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:20:14.824555 3819 raft_consensus.cc:3037] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:14.825541 3819 raft_consensus.cc:504] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:14.825850 3819 leader_election.cc:288] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [CANDIDATE]: Term 1 election: Requested vote from peers cc6424bc538b4127865238407c5d381c (127.31.204.193:40473), b036ffb2a7a34ac38869ce9670afeba8 (127.31.204.194:41177)
I20260810 06:20:14.826225 3380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86eb1274ac8d47d5b83d63953a3be22e" candidate_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b036ffb2a7a34ac38869ce9670afeba8"
I20260810 06:20:14.826234 3152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86eb1274ac8d47d5b83d63953a3be22e" candidate_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc6424bc538b4127865238407c5d381c"
I20260810 06:20:14.826341 3380 raft_consensus.cc:3037] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:14.826360 3152 raft_consensus.cc:3037] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:14.827165 3380 raft_consensus.cc:2454] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e168a43cb2dc432da51e7d1f0fce85f6 in term 1.
I20260810 06:20:14.827452 3513 leader_election.cc:302] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b036ffb2a7a34ac38869ce9670afeba8, e168a43cb2dc432da51e7d1f0fce85f6; no voters:
I20260810 06:20:14.827625 3819 raft_consensus.cc:2781] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:14.827644 3152 raft_consensus.cc:2454] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e168a43cb2dc432da51e7d1f0fce85f6 in term 1.
I20260810 06:20:14.827694 3819 raft_consensus.cc:686] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 LEADER]: Becoming Leader. State: Replica: e168a43cb2dc432da51e7d1f0fce85f6, State: Running, Role: LEADER
I20260810 06:20:14.827821 3819 consensus_queue.cc:237] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:14.829725 2890 catalog_manager.cc:5495] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e168a43cb2dc432da51e7d1f0fce85f6 (127.31.204.195). New cstate: current_term: 1 leader_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } health_report { overall_health: HEALTHY } } }
I20260810 06:20:14.860234 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.857020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44674 (local address 127.31.204.195:37755)
0810 06:20:14.857145 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:14.857151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:14.857167 (+ 16us) server_negotiation.cc:407] Connection header received
0810 06:20:14.857277 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:14.857281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:14.857340 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:14.857418 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:14.857604 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.858419 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.859194 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.859526 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.859659 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.859817 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:14.859902 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:14.859971 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:14.860034 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 06:20:14.860092 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 06:20:14.868158 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.865507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:59189 (local address 127.31.204.195:37755)
0810 06:20:14.865599 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:14.865603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:14.865615 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:20:14.865742 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:14.865745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:14.865783 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:14.865838 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:14.866051 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.866540 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.867290 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.867499 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.867582 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.867791 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:14.867852 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:14.867908 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:14.867990 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 06:20:14.868026 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 06:20:14.868158 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:14.865424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:37755 (local address 127.31.204.195:59189)
0810 06:20:14.865562 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:14.865577 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:14.865652 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:14.865857 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:14.865860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:14.865866 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:20:14.865941 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.865946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.866641 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.866644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.867180 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:14.867198 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:14.867597 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:14.867600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:14.867677 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:14.867700 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:20:14.867922 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:20:14.867928 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 06:20:14.867982 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 06:20:14.868049 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 06:20:14.872910 3380 raft_consensus.cc:1261] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8 [term 1 FOLLOWER]: Refusing update from remote peer e168a43cb2dc432da51e7d1f0fce85f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:14.872951 3152 raft_consensus.cc:1261] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c [term 1 FOLLOWER]: Refusing update from remote peer e168a43cb2dc432da51e7d1f0fce85f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:14.873260 3775 consensus_queue.cc:1035] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:14.873399 3751 consensus_queue.cc:1035] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 06:20:14.887140 3818 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 06:20:16.437975 2890 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:16.438163 2890 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:16.440315 2890 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b: Sending DeleteTablet for 3 replicas of tablet 86eb1274ac8d47d5b83d63953a3be22e
I20260810 06:20:16.440805 3122 tablet_service.cc:1514] Processing DeleteTablet for tablet 86eb1274ac8d47d5b83d63953a3be22e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:16 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:44112
I20260810 06:20:16.440858 3588 tablet_service.cc:1514] Processing DeleteTablet for tablet 86eb1274ac8d47d5b83d63953a3be22e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:16 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:44654
I20260810 06:20:16.440896 3351 tablet_service.cc:1514] Processing DeleteTablet for tablet 86eb1274ac8d47d5b83d63953a3be22e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:16 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:50752
I20260810 06:20:16.441203 3873 tablet_replica.cc:330] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c: stopping tablet replica
I20260810 06:20:16.441402 3873 raft_consensus.cc:2229] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:16.441524 3873 raft_consensus.cc:2258] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:16.442116 3873 ts_tablet_manager.cc:1905] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:16.444398 3876 tablet_replica.cc:330] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6: stopping tablet replica
I20260810 06:20:16.444528 3876 raft_consensus.cc:2229] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:20:16.444806 3873 ts_tablet_manager.cc:1918] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 06:20:16.444839 3876 raft_consensus.cc:2258] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:16.444892 3873 log.cc:1196] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/wal/wals/86eb1274ac8d47d5b83d63953a3be22e
I20260810 06:20:16.445063 3879 tablet_replica.cc:330] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8: stopping tablet replica
I20260810 06:20:16.445192 3873 ts_tablet_manager.cc:1939] T 86eb1274ac8d47d5b83d63953a3be22e P cc6424bc538b4127865238407c5d381c: Deleting consensus metadata
I20260810 06:20:16.445222 3879 raft_consensus.cc:2229] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:16.445374 3879 raft_consensus.cc:2258] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:16.445407 3876 ts_tablet_manager.cc:1905] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:16.445855 2863 catalog_manager.cc:4841] TS cc6424bc538b4127865238407c5d381c (127.31.204.193:40473): tablet 86eb1274ac8d47d5b83d63953a3be22e (table test-table [id=f2e8591512a640cfbe35f6e0844453aa]) successfully deleted
I20260810 06:20:16.445999 3879 ts_tablet_manager.cc:1905] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:16.447731 3876 ts_tablet_manager.cc:1918] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 06:20:16.447868 3876 log.cc:1196] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/86eb1274ac8d47d5b83d63953a3be22e
I20260810 06:20:16.448184 3876 ts_tablet_manager.cc:1939] T 86eb1274ac8d47d5b83d63953a3be22e P e168a43cb2dc432da51e7d1f0fce85f6: Deleting consensus metadata
I20260810 06:20:16.448684 3879 ts_tablet_manager.cc:1918] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 06:20:16.448711 2863 catalog_manager.cc:4841] TS e168a43cb2dc432da51e7d1f0fce85f6 (127.31.204.195:37755): tablet 86eb1274ac8d47d5b83d63953a3be22e (table test-table [id=f2e8591512a640cfbe35f6e0844453aa]) successfully deleted
I20260810 06:20:16.448760 3879 log.cc:1196] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/86eb1274ac8d47d5b83d63953a3be22e
I20260810 06:20:16.448997 3879 ts_tablet_manager.cc:1939] T 86eb1274ac8d47d5b83d63953a3be22e P b036ffb2a7a34ac38869ce9670afeba8: Deleting consensus metadata
I20260810 06:20:16.449548 2865 catalog_manager.cc:4841] TS b036ffb2a7a34ac38869ce9670afeba8 (127.31.204.194:41177): tablet 86eb1274ac8d47d5b83d63953a3be22e (table test-table [id=f2e8591512a640cfbe35f6e0844453aa]) successfully deleted
Aug 10 06:20:16 dist-test-slave-q9h9 krb5kdc[2821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 06:20:16 dist-test-slave-q9h9 krb5kdc[2821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:16.468645 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:16.461147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43168 (local address 127.31.204.254:41717)
0810 06:20:16.461405 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:16.461411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:16.461428 (+ 17us) server_negotiation.cc:407] Connection header received
0810 06:20:16.461545 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:16.461547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:16.461586 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:16.461658 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:16.462307 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:16.462839 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:16.463576 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:16.463784 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:16.463849 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:16.466201 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:16.466204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:16.466206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:16.466229 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:16.467438 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:16.467787 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:16.467790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:16.467791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:16.467860 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:16.468041 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:16.468043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:16.468044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:16.468197 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:16.468313 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:16.468436 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 06:20:16.468525 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":101,"threads_started":1}
I20260810 06:20:16.470904 2890 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:20:16.471305 2890 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:20:16.475663 3122 tablet_service.cc:1467] Processing CreateTablet for tablet 696be363fdc542d986c841f845cc9440 (DEFAULT_TABLE table=test-table [id=a61ed69b917d4ec5986917f3b46a5085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:16.475906 3351 tablet_service.cc:1467] Processing CreateTablet for tablet 696be363fdc542d986c841f845cc9440 (DEFAULT_TABLE table=test-table [id=a61ed69b917d4ec5986917f3b46a5085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:16.475973 3122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 696be363fdc542d986c841f845cc9440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:16.476171 3351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 696be363fdc542d986c841f845cc9440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:16.477907 3891 tablet_bootstrap.cc:492] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c: Bootstrap starting.
I20260810 06:20:16.478099 3892 tablet_bootstrap.cc:492] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8: Bootstrap starting.
I20260810 06:20:16.478235 3588 tablet_service.cc:1467] Processing CreateTablet for tablet 696be363fdc542d986c841f845cc9440 (DEFAULT_TABLE table=test-table [id=a61ed69b917d4ec5986917f3b46a5085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:16.478504 3588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 696be363fdc542d986c841f845cc9440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:16.478920 3892 tablet_bootstrap.cc:654] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:16.479208 3891 tablet_bootstrap.cc:654] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:16.479941 3892 tablet_bootstrap.cc:492] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8: No bootstrap required, opened a new log
I20260810 06:20:16.480013 3892 ts_tablet_manager.cc:1397] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 06:20:16.480331 3891 tablet_bootstrap.cc:492] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c: No bootstrap required, opened a new log
I20260810 06:20:16.480409 3891 ts_tablet_manager.cc:1397] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 06:20:16.480386 3892 raft_consensus.cc:348] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:16.480461 3892 raft_consensus.cc:374] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:16.480482 3892 raft_consensus.cc:729] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b036ffb2a7a34ac38869ce9670afeba8, State: Initialized, Role: FOLLOWER
I20260810 06:20:16.480574 3892 consensus_queue.cc:260] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:16.480849 3896 tablet_bootstrap.cc:492] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6: Bootstrap starting.
I20260810 06:20:16.480931 3892 ts_tablet_manager.cc:1428] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 06:20:16.481534 3891 raft_consensus.cc:348] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:16.481693 3891 raft_consensus.cc:374] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:16.481717 3896 tablet_bootstrap.cc:654] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:16.481743 3891 raft_consensus.cc:729] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc6424bc538b4127865238407c5d381c, State: Initialized, Role: FOLLOWER
I20260810 06:20:16.481871 3891 consensus_queue.cc:260] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:16.482275 3891 ts_tablet_manager.cc:1428] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:16.483234 3896 tablet_bootstrap.cc:492] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6: No bootstrap required, opened a new log
I20260810 06:20:16.483328 3896 ts_tablet_manager.cc:1397] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 06:20:16.483670 3896 raft_consensus.cc:348] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:16.483790 3896 raft_consensus.cc:374] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:16.483857 3896 raft_consensus.cc:729] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e168a43cb2dc432da51e7d1f0fce85f6, State: Initialized, Role: FOLLOWER
I20260810 06:20:16.483985 3896 consensus_queue.cc:260] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:16.484227 3896 ts_tablet_manager.cc:1428] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:16.694645 3900 raft_consensus.cc:482] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:20:16.694826 3900 raft_consensus.cc:504] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:16.695247 3900 leader_election.cc:288] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b036ffb2a7a34ac38869ce9670afeba8 (127.31.204.194:41177), e168a43cb2dc432da51e7d1f0fce85f6 (127.31.204.195:37755)
I20260810 06:20:16.695701 3613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696be363fdc542d986c841f845cc9440" candidate_uuid: "cc6424bc538b4127865238407c5d381c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" is_pre_election: true
I20260810 06:20:16.695716 3380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696be363fdc542d986c841f845cc9440" candidate_uuid: "cc6424bc538b4127865238407c5d381c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b036ffb2a7a34ac38869ce9670afeba8" is_pre_election: true
I20260810 06:20:16.695845 3613 raft_consensus.cc:2454] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc6424bc538b4127865238407c5d381c in term 0.
I20260810 06:20:16.695851 3380 raft_consensus.cc:2454] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc6424bc538b4127865238407c5d381c in term 0.
I20260810 06:20:16.696180 3045 leader_election.cc:302] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc6424bc538b4127865238407c5d381c, e168a43cb2dc432da51e7d1f0fce85f6; no voters:
I20260810 06:20:16.696350 3900 raft_consensus.cc:2781] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:20:16.696422 3900 raft_consensus.cc:482] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:20:16.696466 3900 raft_consensus.cc:3037] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:16.697474 3900 raft_consensus.cc:504] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:16.697793 3900 leader_election.cc:288] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [CANDIDATE]: Term 1 election: Requested vote from peers b036ffb2a7a34ac38869ce9670afeba8 (127.31.204.194:41177), e168a43cb2dc432da51e7d1f0fce85f6 (127.31.204.195:37755)
I20260810 06:20:16.698140 3613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696be363fdc542d986c841f845cc9440" candidate_uuid: "cc6424bc538b4127865238407c5d381c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e168a43cb2dc432da51e7d1f0fce85f6"
I20260810 06:20:16.698138 3380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696be363fdc542d986c841f845cc9440" candidate_uuid: "cc6424bc538b4127865238407c5d381c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b036ffb2a7a34ac38869ce9670afeba8"
I20260810 06:20:16.698237 3380 raft_consensus.cc:3037] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:16.698239 3613 raft_consensus.cc:3037] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:16.699083 3380 raft_consensus.cc:2454] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc6424bc538b4127865238407c5d381c in term 1.
I20260810 06:20:16.699146 3613 raft_consensus.cc:2454] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc6424bc538b4127865238407c5d381c in term 1.
I20260810 06:20:16.699332 3047 leader_election.cc:302] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b036ffb2a7a34ac38869ce9670afeba8, cc6424bc538b4127865238407c5d381c; no voters:
I20260810 06:20:16.699483 3900 raft_consensus.cc:2781] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:16.699694 3900 raft_consensus.cc:686] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [term 1 LEADER]: Becoming Leader. State: Replica: cc6424bc538b4127865238407c5d381c, State: Running, Role: LEADER
I20260810 06:20:16.699824 3900 consensus_queue.cc:237] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:16.701365 2892 catalog_manager.cc:5495] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c reported cstate change: term changed from 0 to 1, leader changed from <none> to cc6424bc538b4127865238407c5d381c (127.31.204.193). New cstate: current_term: 1 leader_uuid: "cc6424bc538b4127865238407c5d381c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } health_report { overall_health: UNKNOWN } } }
I20260810 06:20:16.716840 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:16.713924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44132 (local address 127.31.204.193:40473)
0810 06:20:16.714182 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:16.714189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:16.714200 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:20:16.714235 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:16.714237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:16.714276 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:16.714343 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:16.714539 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:16.715085 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:16.715935 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:16.716209 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:16.716311 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:16.716485 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:16.716548 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:16.716593 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:16.716656 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 06:20:16.716696 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":123,"threads_started":1}
I20260810 06:20:16.722208 3613 raft_consensus.cc:1261] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 FOLLOWER]: Refusing update from remote peer cc6424bc538b4127865238407c5d381c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:16.722561 3904 consensus_queue.cc:1035] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:16.722617 3380 raft_consensus.cc:1261] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8 [term 1 FOLLOWER]: Refusing update from remote peer cc6424bc538b4127865238407c5d381c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:16.722957 3904 consensus_queue.cc:1035] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:16.752949 2892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:16.753095 2892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:16.754899 2892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b: Sending DeleteTablet for 3 replicas of tablet 696be363fdc542d986c841f845cc9440
I20260810 06:20:16.755353 3122 tablet_service.cc:1514] Processing DeleteTablet for tablet 696be363fdc542d986c841f845cc9440 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:16 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:44112
I20260810 06:20:16.755412 3351 tablet_service.cc:1514] Processing DeleteTablet for tablet 696be363fdc542d986c841f845cc9440 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:16 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:50752
I20260810 06:20:16.755537 3873 tablet_replica.cc:330] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c: stopping tablet replica
I20260810 06:20:16.755556 3879 tablet_replica.cc:330] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8: stopping tablet replica
I20260810 06:20:16.755648 3879 raft_consensus.cc:2229] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:16.755654 3873 raft_consensus.cc:2229] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:20:16.755761 3879 raft_consensus.cc:2258] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:16.755827 3873 raft_consensus.cc:2258] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:16.756017 2892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 06:20:16.756126 3588 tablet_service.cc:1514] Processing DeleteTablet for tablet 696be363fdc542d986c841f845cc9440 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:16 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:44654
I20260810 06:20:16.756315 3876 tablet_replica.cc:330] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6: stopping tablet replica
W20260810 06:20:16.756407 2892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:20:16.756422 3876 raft_consensus.cc:2229] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:16.756525 3876 raft_consensus.cc:2258] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:16.756531 3879 ts_tablet_manager.cc:1905] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:16.756708 3873 ts_tablet_manager.cc:1905] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:16.759025 3879 ts_tablet_manager.cc:1918] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:20:16.759097 3879 log.cc:1196] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/696be363fdc542d986c841f845cc9440
I20260810 06:20:16.759135 3873 ts_tablet_manager.cc:1918] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:20:16.759204 3873 log.cc:1196] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-0/wal/wals/696be363fdc542d986c841f845cc9440
I20260810 06:20:16.759356 3879 ts_tablet_manager.cc:1939] T 696be363fdc542d986c841f845cc9440 P b036ffb2a7a34ac38869ce9670afeba8: Deleting consensus metadata
I20260810 06:20:16.759423 3873 ts_tablet_manager.cc:1939] T 696be363fdc542d986c841f845cc9440 P cc6424bc538b4127865238407c5d381c: Deleting consensus metadata
I20260810 06:20:16.759963 2865 catalog_manager.cc:4841] TS b036ffb2a7a34ac38869ce9670afeba8 (127.31.204.194:41177): tablet 696be363fdc542d986c841f845cc9440 (table test-table [id=a61ed69b917d4ec5986917f3b46a5085]) successfully deleted
I20260810 06:20:16.760039 2863 catalog_manager.cc:4841] TS cc6424bc538b4127865238407c5d381c (127.31.204.193:40473): tablet 696be363fdc542d986c841f845cc9440 (table test-table [id=a61ed69b917d4ec5986917f3b46a5085]) successfully deleted
I20260810 06:20:16.760455 3876 ts_tablet_manager.cc:1905] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:16.762279 3122 tablet_service.cc:1467] Processing CreateTablet for tablet 231d87659fe849748fa0f2b9bae46c30 (DEFAULT_TABLE table=test-table [id=a9b007b8740d4550b7a8da89c56b4502]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:16.762351 3351 tablet_service.cc:1467] Processing CreateTablet for tablet 231d87659fe849748fa0f2b9bae46c30 (DEFAULT_TABLE table=test-table [id=a9b007b8740d4550b7a8da89c56b4502]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:16.762465 3588 tablet_service.cc:1467] Processing CreateTablet for tablet 231d87659fe849748fa0f2b9bae46c30 (DEFAULT_TABLE table=test-table [id=a9b007b8740d4550b7a8da89c56b4502]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:16.762531 3122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 231d87659fe849748fa0f2b9bae46c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:16.762573 3351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 231d87659fe849748fa0f2b9bae46c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:16.762668 3588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 231d87659fe849748fa0f2b9bae46c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:16.762966 3876 ts_tablet_manager.cc:1918] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:20:16.763026 3876 log.cc:1196] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/696be363fdc542d986c841f845cc9440
I20260810 06:20:16.763258 3876 ts_tablet_manager.cc:1939] T 696be363fdc542d986c841f845cc9440 P e168a43cb2dc432da51e7d1f0fce85f6: Deleting consensus metadata
I20260810 06:20:16.764554 3891 tablet_bootstrap.cc:492] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c: Bootstrap starting.
I20260810 06:20:16.764642 3892 tablet_bootstrap.cc:492] T 231d87659fe849748fa0f2b9bae46c30 P b036ffb2a7a34ac38869ce9670afeba8: Bootstrap starting.
I20260810 06:20:16.764865 2863 catalog_manager.cc:4841] TS e168a43cb2dc432da51e7d1f0fce85f6 (127.31.204.195:37755): tablet 696be363fdc542d986c841f845cc9440 (table test-table [id=a61ed69b917d4ec5986917f3b46a5085]) successfully deleted
I20260810 06:20:16.765178 3896 tablet_bootstrap.cc:492] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6: Bootstrap starting.
I20260810 06:20:16.765341 3891 tablet_bootstrap.cc:654] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:16.765870 3892 tablet_bootstrap.cc:654] T 231d87659fe849748fa0f2b9bae46c30 P b036ffb2a7a34ac38869ce9670afeba8: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:16.765975 3896 tablet_bootstrap.cc:654] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:16.766395 3891 tablet_bootstrap.cc:492] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c: No bootstrap required, opened a new log
I20260810 06:20:16.766481 3891 ts_tablet_manager.cc:1397] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 06:20:16.766822 3891 raft_consensus.cc:348] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:16.766903 3891 raft_consensus.cc:374] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:16.766922 3891 raft_consensus.cc:729] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc6424bc538b4127865238407c5d381c, State: Initialized, Role: FOLLOWER
I20260810 06:20:16.767027 3892 tablet_bootstrap.cc:492] T 231d87659fe849748fa0f2b9bae46c30 P b036ffb2a7a34ac38869ce9670afeba8: No bootstrap required, opened a new log
I20260810 06:20:16.767017 3891 consensus_queue.cc:260] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:16.767100 3892 ts_tablet_manager.cc:1397] T 231d87659fe849748fa0f2b9bae46c30 P b036ffb2a7a34ac38869ce9670afeba8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:16.767213 3896 tablet_bootstrap.cc:492] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6: No bootstrap required, opened a new log
I20260810 06:20:16.767263 3891 ts_tablet_manager.cc:1428] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 06:20:16.767279 3896 ts_tablet_manager.cc:1397] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:16.767607 3896 raft_consensus.cc:348] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:16.767709 3896 raft_consensus.cc:374] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:16.767742 3896 raft_consensus.cc:729] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e168a43cb2dc432da51e7d1f0fce85f6, State: Initialized, Role: FOLLOWER
I20260810 06:20:16.767864 3896 consensus_queue.cc:260] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:16.768105 3896 ts_tablet_manager.cc:1428] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 06:20:16.769533 3892 raft_consensus.cc:348] T 231d87659fe849748fa0f2b9bae46c30 P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:16.769651 3892 raft_consensus.cc:374] T 231d87659fe849748fa0f2b9bae46c30 P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:16.769683 3892 raft_consensus.cc:729] T 231d87659fe849748fa0f2b9bae46c30 P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b036ffb2a7a34ac38869ce9670afeba8, State: Initialized, Role: FOLLOWER
I20260810 06:20:16.769785 3892 consensus_queue.cc:260] T 231d87659fe849748fa0f2b9bae46c30 P b036ffb2a7a34ac38869ce9670afeba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:16.770080 3892 ts_tablet_manager.cc:1428] T 231d87659fe849748fa0f2b9bae46c30 P b036ffb2a7a34ac38869ce9670afeba8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260810 06:20:16.874449 3919 raft_consensus.cc:482] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:20:16.874599 3919 raft_consensus.cc:504] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:16.875083 3919 leader_election.cc:288] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b036ffb2a7a34ac38869ce9670afeba8 (127.31.204.194:41177), e168a43cb2dc432da51e7d1f0fce85f6 (127.31.204.195:37755)
I20260810 06:20:16.875440 3380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231d87659fe849748fa0f2b9bae46c30" candidate_uuid: "cc6424bc538b4127865238407c5d381c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b036ffb2a7a34ac38869ce9670afeba8" is_pre_election: true
I20260810 06:20:16.875484 3613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231d87659fe849748fa0f2b9bae46c30" candidate_uuid: "cc6424bc538b4127865238407c5d381c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" is_pre_election: true
I20260810 06:20:16.875612 3380 raft_consensus.cc:2454] T 231d87659fe849748fa0f2b9bae46c30 P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc6424bc538b4127865238407c5d381c in term 0.
I20260810 06:20:16.875658 3613 raft_consensus.cc:2454] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc6424bc538b4127865238407c5d381c in term 0.
I20260810 06:20:16.875954 3047 leader_election.cc:302] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b036ffb2a7a34ac38869ce9670afeba8, cc6424bc538b4127865238407c5d381c; no voters:
I20260810 06:20:16.876127 3919 raft_consensus.cc:2781] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:20:16.876201 3919 raft_consensus.cc:482] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:20:16.876247 3919 raft_consensus.cc:3037] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:16.877321 3919 raft_consensus.cc:504] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:16.877631 3919 leader_election.cc:288] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c [CANDIDATE]: Term 1 election: Requested vote from peers b036ffb2a7a34ac38869ce9670afeba8 (127.31.204.194:41177), e168a43cb2dc432da51e7d1f0fce85f6 (127.31.204.195:37755)
I20260810 06:20:16.877976 3380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231d87659fe849748fa0f2b9bae46c30" candidate_uuid: "cc6424bc538b4127865238407c5d381c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b036ffb2a7a34ac38869ce9670afeba8"
I20260810 06:20:16.877982 3613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231d87659fe849748fa0f2b9bae46c30" candidate_uuid: "cc6424bc538b4127865238407c5d381c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e168a43cb2dc432da51e7d1f0fce85f6"
I20260810 06:20:16.878113 3380 raft_consensus.cc:3037] T 231d87659fe849748fa0f2b9bae46c30 P b036ffb2a7a34ac38869ce9670afeba8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:16.878150 3613 raft_consensus.cc:3037] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:16.879055 3380 raft_consensus.cc:2454] T 231d87659fe849748fa0f2b9bae46c30 P b036ffb2a7a34ac38869ce9670afeba8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc6424bc538b4127865238407c5d381c in term 1.
I20260810 06:20:16.879235 3613 raft_consensus.cc:2454] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc6424bc538b4127865238407c5d381c in term 1.
I20260810 06:20:16.879318 3047 leader_election.cc:302] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b036ffb2a7a34ac38869ce9670afeba8, cc6424bc538b4127865238407c5d381c; no voters:
I20260810 06:20:16.879516 3919 raft_consensus.cc:2781] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:16.879621 3919 raft_consensus.cc:686] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c [term 1 LEADER]: Becoming Leader. State: Replica: cc6424bc538b4127865238407c5d381c, State: Running, Role: LEADER
I20260810 06:20:16.879773 3919 consensus_queue.cc:237] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } }
I20260810 06:20:16.881422 2891 catalog_manager.cc:5495] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c reported cstate change: term changed from 0 to 1, leader changed from <none> to cc6424bc538b4127865238407c5d381c (127.31.204.193). New cstate: current_term: 1 leader_uuid: "cc6424bc538b4127865238407c5d381c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6424bc538b4127865238407c5d381c" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 } health_report { overall_health: UNKNOWN } } }
I20260810 06:20:16.922588 3380 raft_consensus.cc:1261] T 231d87659fe849748fa0f2b9bae46c30 P b036ffb2a7a34ac38869ce9670afeba8 [term 1 FOLLOWER]: Refusing update from remote peer cc6424bc538b4127865238407c5d381c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:16.922797 3613 raft_consensus.cc:1261] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 FOLLOWER]: Refusing update from remote peer cc6424bc538b4127865238407c5d381c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:16.923149 3919 consensus_queue.cc:1035] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e168a43cb2dc432da51e7d1f0fce85f6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:16.923326 3904 consensus_queue.cc:1035] T 231d87659fe849748fa0f2b9bae46c30 P cc6424bc538b4127865238407c5d381c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b036ffb2a7a34ac38869ce9670afeba8" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 06:20:16.935223 3850 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 06:20:18.485409 2891 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:18.485554 2891 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:18.487733 2891 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 021eb73e52824092a1965bbe3b52e36b: Sending DeleteTablet for 3 replicas of tablet 231d87659fe849748fa0f2b9bae46c30
I20260810 06:20:18.488212 3351 tablet_service.cc:1514] Processing DeleteTablet for tablet 231d87659fe849748fa0f2b9bae46c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:18 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:50752
I20260810 06:20:18.488219 3122 tablet_service.cc:1514] Processing DeleteTablet for tablet 231d87659fe849748fa0f2b9bae46c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:18 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:44112
I20260810 06:20:18.488260 3587 tablet_service.cc:1514] Processing DeleteTablet for tablet 231d87659fe849748fa0f2b9bae46c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:18 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:44654
I20260810 06:20:18.488639 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 3012
I20260810 06:20:18.495788 3966 tablet_replica.cc:330] T 231d87659fe849748fa0f2b9bae46c30 P b036ffb2a7a34ac38869ce9670afeba8: stopping tablet replica
I20260810 06:20:18.495923 3966 raft_consensus.cc:2229] T 231d87659fe849748fa0f2b9bae46c30 P b036ffb2a7a34ac38869ce9670afeba8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:18.496030 3966 raft_consensus.cc:2258] T 231d87659fe849748fa0f2b9bae46c30 P b036ffb2a7a34ac38869ce9670afeba8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:18.496240 3967 tablet_replica.cc:330] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6: stopping tablet replica
I20260810 06:20:18.496376 3967 raft_consensus.cc:2229] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:18.496479 3967 raft_consensus.cc:2258] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:18.496951 3966 ts_tablet_manager.cc:1905] T 231d87659fe849748fa0f2b9bae46c30 P b036ffb2a7a34ac38869ce9670afeba8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:18.498881 3967 ts_tablet_manager.cc:1905] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 06:20:18.506630 2863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.204.193:40473 (error 108)
I20260810 06:20:18.506863 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 3246
I20260810 06:20:18.516733 3971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:18.516339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:40473 (local address 127.0.0.1:47826)
0810 06:20:18.516569 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:18.516649 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:40473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":78,"threads_started":1}
W20260810 06:20:18.516878 2863 catalog_manager.cc:4584] TS cc6424bc538b4127865238407c5d381c (127.31.204.193:40473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 231d87659fe849748fa0f2b9bae46c30: Network error: Client connection negotiation failed: client connection to 127.31.204.193:40473: connect: Connection refused (error 111)
I20260810 06:20:18.556558 3967 ts_tablet_manager.cc:1918] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 06:20:18.556725 3967 log.cc:1196] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/231d87659fe849748fa0f2b9bae46c30
I20260810 06:20:18.557010 3967 ts_tablet_manager.cc:1939] T 231d87659fe849748fa0f2b9bae46c30 P e168a43cb2dc432da51e7d1f0fce85f6: Deleting consensus metadata
I20260810 06:20:18.557363 3971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:18.557023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:41177 (local address 127.0.0.1:40032)
0810 06:20:18.557219 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:18.557275 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.194:41177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260810 06:20:18.557528 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 3477
I20260810 06:20:18.569864 3971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:18.569590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:40473 (local address 127.0.0.1:47842)
0810 06:20:18.569734 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:18.569788 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:40473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 06:20:18.570521 3971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:18.570347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:40473 (local address 127.0.0.1:47852)
0810 06:20:18.570426 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:18.570468 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:40473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 06:20:18.586246 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 2836
2026-08-10T06:20:18Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6524 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:20:19 dist-test-slave-q9h9 krb5kdc[3974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:20:19 dist-test-slave-q9h9 krb5kdc[3974](info): set up 2 sockets
Aug 10 06:20:19 dist-test-slave-q9h9 krb5kdc[3974](info): commencing operation
krb5kdc: starting...
W20260810 06:20:20.710294 32563 mini_kdc.cc:121] Time spent starting KDC: real 2.092s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:20:20 dist-test-slave-q9h9 krb5kdc[3974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:20:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:20:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:20:20.823946 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:39105
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:42547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:39105
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:20.932367 3996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:20.932597 3996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:20.932638 3996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:20.936570 3996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:20.936621 3996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:20.936640 3996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:20.936657 3996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:20.936687 3996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:20:20.941766 3996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:42547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:39105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:39105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:20.942775 3996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:20.943744 3996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:20:20.951723 3996 server_base.cc:1034] running on GCE node
W20260810 06:20:20.951684 4005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:20.951663 4010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:20.951689 4008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:20.952471 3996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:20.953439 3996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:20.954589 3996 hybrid_clock.cc:648] HybridClock initialized: now 1786342820954579 us; error 26 us; skew 500 ppm
Aug 10 06:20:20 dist-test-slave-q9h9 krb5kdc[3974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342820, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:20.957113 3996 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:20:20.957959 3996 webserver.cc:460] Webserver started at http://127.31.204.254:41027/ using document root <none> and password file <none>
I20260810 06:20:20.958474 3996 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:20.958540 3996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:20.958726 3996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:20.960641 3996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "5e50c755c8034863ae8b4e5343e2562d"
format_stamp: "Formatted at 2026-08-10 06:20:20 on dist-test-slave-q9h9"
server_key: "eeb0990f36dc99df58fe06b65a8a42dd"
server_key_iv: "fd76c0411dd45b2179ee9cf294d5e0c5"
server_key_version: "encryptionkey@0"
I20260810 06:20:20.961159 3996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "5e50c755c8034863ae8b4e5343e2562d"
format_stamp: "Formatted at 2026-08-10 06:20:20 on dist-test-slave-q9h9"
server_key: "eeb0990f36dc99df58fe06b65a8a42dd"
server_key_iv: "fd76c0411dd45b2179ee9cf294d5e0c5"
server_key_version: "encryptionkey@0"
I20260810 06:20:20.964643 3996 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 06:20:20.966858 4019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:20.968039 3996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260810 06:20:20.968148 3996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "5e50c755c8034863ae8b4e5343e2562d"
format_stamp: "Formatted at 2026-08-10 06:20:20 on dist-test-slave-q9h9"
server_key: "eeb0990f36dc99df58fe06b65a8a42dd"
server_key_iv: "fd76c0411dd45b2179ee9cf294d5e0c5"
server_key_version: "encryptionkey@0"
I20260810 06:20:20.968233 3996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:20.976719 3996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:20.979303 3996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:20.979455 3996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:20.985713 3996 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:39105
I20260810 06:20:20.985766 4096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:39105 every 8 connection(s)
I20260810 06:20:20.986593 3996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:20:20.989094 4097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:20.989411 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 3996
I20260810 06:20:20.989509 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:20:20.989739 32563 external_mini_cluster.cc:1428] Setting key c49ab3251cf6b3f572d42c9c70a068f7
I20260810 06:20:20.995788 4097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d: Bootstrap starting.
I20260810 06:20:20.998266 4097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d: Neither blocks nor log segments found. Creating new log.
Aug 10 06:20:20 dist-test-slave-q9h9 krb5kdc[3974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:20.999148 4097 log.cc:824] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:21.000972 4097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d: No bootstrap required, opened a new log
I20260810 06:20:21.003590 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:20.990995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.31.204.254:39105)
0810 06:20:20.991405 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:20.991423 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:20.991465 (+ 42us) server_negotiation.cc:407] Connection header received
0810 06:20:20.992176 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:20.992199 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:20.992590 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:20.992947 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:20.993561 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:20.995104 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:20.996058 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:20.996719 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:20.996878 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:20.999436 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:20.999441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:20.999453 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:20.999479 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:21.000947 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:21.001422 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:21.001427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:21.001431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:21.001534 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:21.001815 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:21.001817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:21.001818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:21.002217 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:21.002372 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:21.002640 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 06:20:21.002875 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":108,"threads_started":1}
I20260810 06:20:21.004812 4097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e50c755c8034863ae8b4e5343e2562d" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 39105 } }
I20260810 06:20:21.005075 4097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:21.005133 4097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e50c755c8034863ae8b4e5343e2562d, State: Initialized, Role: FOLLOWER
I20260810 06:20:21.005558 4097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e50c755c8034863ae8b4e5343e2562d" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 39105 } }
I20260810 06:20:21.005692 4097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:20:21.005759 4097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:20:21.005852 4097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:21.006708 4097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e50c755c8034863ae8b4e5343e2562d" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 39105 } }
I20260810 06:20:21.007012 4097 leader_election.cc:302] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e50c755c8034863ae8b4e5343e2562d; no voters:
I20260810 06:20:21.007329 4097 leader_election.cc:288] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:20:21.007421 4103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:21.007623 4103 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d [term 1 LEADER]: Becoming Leader. State: Replica: 5e50c755c8034863ae8b4e5343e2562d, State: Running, Role: LEADER
I20260810 06:20:21.007908 4103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e50c755c8034863ae8b4e5343e2562d" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 39105 } }
I20260810 06:20:21.008494 4097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:21.008910 4103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e50c755c8034863ae8b4e5343e2562d. Latest consensus state: current_term: 1 leader_uuid: "5e50c755c8034863ae8b4e5343e2562d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e50c755c8034863ae8b4e5343e2562d" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 39105 } } }
I20260810 06:20:21.009059 4103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:21.009641 4104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e50c755c8034863ae8b4e5343e2562d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e50c755c8034863ae8b4e5343e2562d" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 39105 } } }
I20260810 06:20:21.009814 4104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:21.009728 4112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:21.012944 4112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:21.018280 4112 catalog_manager.cc:1347] Generated new cluster ID: 6f8135aa19314db28b2e3bb9b7bb387d
I20260810 06:20:21.018343 4112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:21.033131 4112 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:20:21.034327 4112 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:21.041482 4112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e50c755c8034863ae8b4e5343e2562d: Generated new TSK 0
I20260810 06:20:21.042191 4112 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
I20260810 06:20:21.070384 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:39105
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:42547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:21.175969 4140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:21.176185 4140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:21.176225 4140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:21.180059 4140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:21.180105 4140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:21.180171 4140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:20:21.185029 4140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:42547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:39105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:21.185949 4140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:21.186877 4140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:21.195480 4148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:21.195537 4147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:21.195676 4140 server_base.cc:1034] running on GCE node
W20260810 06:20:21.195508 4151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:21.196095 4140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:21.196645 4140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:21.197788 4140 hybrid_clock.cc:648] HybridClock initialized: now 1786342821197766 us; error 34 us; skew 500 ppm
Aug 10 06:20:21 dist-test-slave-q9h9 krb5kdc[3974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342821, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:21.200357 4140 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:20:21.201112 4140 webserver.cc:460] Webserver started at http://127.31.204.193:35127/ using document root <none> and password file <none>
I20260810 06:20:21.201604 4140 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:21.201663 4140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:21.201850 4140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:21.203748 4140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "68ea17c54f54443c9201eb2f3f3377af"
format_stamp: "Formatted at 2026-08-10 06:20:21 on dist-test-slave-q9h9"
server_key: "ac017107fdc75f72a1cb4f350d6e271c"
server_key_iv: "2de436546466515d6ae38923a320c90b"
server_key_version: "encryptionkey@0"
I20260810 06:20:21.204272 4140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "68ea17c54f54443c9201eb2f3f3377af"
format_stamp: "Formatted at 2026-08-10 06:20:21 on dist-test-slave-q9h9"
server_key: "ac017107fdc75f72a1cb4f350d6e271c"
server_key_iv: "2de436546466515d6ae38923a320c90b"
server_key_version: "encryptionkey@0"
I20260810 06:20:21.207652 4140 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:20:21.209959 4162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:21.211122 4140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:21.211210 4140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "68ea17c54f54443c9201eb2f3f3377af"
format_stamp: "Formatted at 2026-08-10 06:20:21 on dist-test-slave-q9h9"
server_key: "ac017107fdc75f72a1cb4f350d6e271c"
server_key_iv: "2de436546466515d6ae38923a320c90b"
server_key_version: "encryptionkey@0"
I20260810 06:20:21.211298 4140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:21.223404 4140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:21.225865 4140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:21.226006 4140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:21.226536 4140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:21.227248 4140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:21.227294 4140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:21.227329 4140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:21.227349 4140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:21.235709 4140 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:43085
I20260810 06:20:21.235742 4348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:43085 every 8 connection(s)
I20260810 06:20:21.236479 4140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
Aug 10 06:20:21 dist-test-slave-q9h9 krb5kdc[3974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342821, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:21.245787 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 4140
I20260810 06:20:21.245893 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:20:21.246191 32563 external_mini_cluster.cc:1428] Setting key 862b5b2dd7ed75588be1651f27440d36
I20260810 06:20:21.249549 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:21.238196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:54727 (local address 127.31.204.254:39105)
0810 06:20:21.238328 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:21.238332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:21.239071 (+ 739us) server_negotiation.cc:407] Connection header received
0810 06:20:21.240319 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:21.240322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:21.240378 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:21.240450 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:21.241505 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:21.242061 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:21.243144 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:21.243391 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:21.243477 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:21.245819 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:21.245822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:21.245824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:21.245853 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:21.247529 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:21.248086 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:21.248090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:21.248092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:21.248196 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:21.248519 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:21.248523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:21.248525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:21.248756 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:21.248854 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:21.249264 (+ 410us) server_negotiation.cc:299] Negotiation successful
0810 06:20:21.249395 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:20:21.250255 4353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:21.238527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:39105 (local address 127.31.204.193:54727)
0810 06:20:21.238938 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:21.238977 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:21.240115 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:21.240611 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:21.240617 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:21.240939 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:21.241359 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:21.241371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:21.242175 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:21.242178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:21.243026 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:21.243035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:21.243525 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:21.243530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:21.243681 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:21.244145 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:21.244165 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:21.245646 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:21.247667 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:21.247673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:21.247686 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:21.247958 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:21.248300 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:21.248302 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:21.248303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:21.248429 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:21.248848 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:21.248852 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:21.249056 (+ 204us) client_negotiation.cc:769] Sending connection context
0810 06:20:21.249196 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 06:20:21.249434 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":115,"threads_started":1}
I20260810 06:20:21.251338 4350 heartbeater.cc:347] Connected to a master server at 127.31.204.254:39105
I20260810 06:20:21.251550 4350 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:21.252007 4350 heartbeater.cc:511] Master 127.31.204.254:39105 requested a full tablet report, sending...
I20260810 06:20:21.253420 4045 ts_manager.cc:194] Registered new tserver with Master: 68ea17c54f54443c9201eb2f3f3377af (127.31.204.193:43085)
I20260810 06:20:21.254551 4045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:54727
WARNING: no policy specified for kudu/127.31.204.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
I20260810 06:20:21.269997 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:39105
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:42547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:21.374038 4357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:21.374257 4357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:21.374315 4357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:21.378172 4357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:21.378221 4357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:21.378307 4357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:20:21.383212 4357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:42547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:39105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:21.384083 4357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:21.384982 4357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:21.393883 4363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:21.393884 4366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:21.393926 4357 server_base.cc:1034] running on GCE node
W20260810 06:20:21.393893 4364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:21.394539 4357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:21.395092 4357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:21.396230 4357 hybrid_clock.cc:648] HybridClock initialized: now 1786342821396209 us; error 34 us; skew 500 ppm
Aug 10 06:20:21 dist-test-slave-q9h9 krb5kdc[3974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342821, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:21.401573 4357 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:20:21.402423 4357 webserver.cc:460] Webserver started at http://127.31.204.194:44035/ using document root <none> and password file <none>
I20260810 06:20:21.402909 4357 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:21.402972 4357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:21.403167 4357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:21.405000 4357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "0d01e6ea7fa44cf0abd881ec000a771c"
format_stamp: "Formatted at 2026-08-10 06:20:21 on dist-test-slave-q9h9"
server_key: "7b026a4a404a9f9ce39a62a6b56800f6"
server_key_iv: "cb92e63e9e22c3e155fb43cc0a1150ca"
server_key_version: "encryptionkey@0"
I20260810 06:20:21.405499 4357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "0d01e6ea7fa44cf0abd881ec000a771c"
format_stamp: "Formatted at 2026-08-10 06:20:21 on dist-test-slave-q9h9"
server_key: "7b026a4a404a9f9ce39a62a6b56800f6"
server_key_iv: "cb92e63e9e22c3e155fb43cc0a1150ca"
server_key_version: "encryptionkey@0"
I20260810 06:20:21.409017 4357 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 06:20:21.411321 4375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:21.412405 4357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:21.412520 4357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "0d01e6ea7fa44cf0abd881ec000a771c"
format_stamp: "Formatted at 2026-08-10 06:20:21 on dist-test-slave-q9h9"
server_key: "7b026a4a404a9f9ce39a62a6b56800f6"
server_key_iv: "cb92e63e9e22c3e155fb43cc0a1150ca"
server_key_version: "encryptionkey@0"
I20260810 06:20:21.412623 4357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:21.428733 4357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:21.431792 4357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:21.431967 4357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:21.432505 4357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:21.433290 4357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:21.433342 4357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:21.433398 4357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:21.433431 4357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:21.442193 4357 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:35899
I20260810 06:20:21.442262 4558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:35899 every 8 connection(s)
I20260810 06:20:21.443034 4357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:20:21.445113 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 4357
I20260810 06:20:21.445212 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:20:21.445540 32563 external_mini_cluster.cc:1428] Setting key 512840606a60b5b6c9b0488c9f422adc
Aug 10 06:20:21 dist-test-slave-q9h9 krb5kdc[3974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342821, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.204.195@KRBTEST.COM; defaulting to no policy
I20260810 06:20:21.456888 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:21.444669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:41857 (local address 127.31.204.254:39105)
0810 06:20:21.444802 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:21.444806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:21.445663 (+ 857us) server_negotiation.cc:407] Connection header received
0810 06:20:21.447258 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:21.447260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:21.447308 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:21.447379 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:21.448719 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:21.449234 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:21.450396 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:21.450661 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:21.450745 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:21.453451 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:21.453454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:21.453456 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:21.453479 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:21.454751 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:21.455361 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:21.455364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:21.455365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:21.455433 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:21.455824 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:21.455826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:21.455827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:21.455982 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:21.456048 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:21.456659 (+ 611us) server_negotiation.cc:299] Negotiation successful
0810 06:20:21.456770 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:20:21.457693 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:21.444993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:39105 (local address 127.31.204.194:41857)
0810 06:20:21.445513 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:21.445566 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:21.447042 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:21.447560 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:21.447568 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:21.448011 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:21.448587 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:21.448604 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:21.449368 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:21.449371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:21.450269 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:21.450280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:21.450819 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:21.450824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:21.451023 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:21.451567 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:21.451595 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:21.453280 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:21.454879 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:21.454886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:21.454900 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:21.455252 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:21.455520 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:21.455523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:21.455526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:21.455721 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:21.456082 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:21.456088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:21.456379 (+ 291us) client_negotiation.cc:769] Sending connection context
0810 06:20:21.456626 (+ 247us) client_negotiation.cc:240] Negotiation successful
0810 06:20:21.456926 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":135,"threads_started":1}
I20260810 06:20:21.458808 4559 heartbeater.cc:347] Connected to a master server at 127.31.204.254:39105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.195@KRBTEST.COM" created.
I20260810 06:20:21.459015 4559 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:21.459498 4559 heartbeater.cc:511] Master 127.31.204.254:39105 requested a full tablet report, sending...
I20260810 06:20:21.460549 4045 ts_manager.cc:194] Registered new tserver with Master: 0d01e6ea7fa44cf0abd881ec000a771c (127.31.204.194:35899)
I20260810 06:20:21.461122 4045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:41857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
I20260810 06:20:21.473024 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:39105
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:42547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:21.590603 4567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:21.590811 4567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:21.590865 4567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:21.594707 4567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:21.594755 4567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:21.594843 4567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:20:21.600003 4567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:42547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:39105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:21.600852 4567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:21.601771 4567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:21.610931 4580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:21.610960 4573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:21.610939 4576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:21.611228 4567 server_base.cc:1034] running on GCE node
I20260810 06:20:21.611596 4567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:21.612169 4567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:21.613323 4567 hybrid_clock.cc:648] HybridClock initialized: now 1786342821613306 us; error 30 us; skew 500 ppm
Aug 10 06:20:21 dist-test-slave-q9h9 krb5kdc[3974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342821, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:21.615935 4567 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:20:21.616771 4567 webserver.cc:460] Webserver started at http://127.31.204.195:41127/ using document root <none> and password file <none>
I20260810 06:20:21.617272 4567 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:21.617336 4567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:21.617592 4567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:21.619570 4567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "da76d2a3616641058aa844a02b66bfdc"
format_stamp: "Formatted at 2026-08-10 06:20:21 on dist-test-slave-q9h9"
server_key: "4af1a148e6f1efe484e053b549ec83c3"
server_key_iv: "18fcf41594691b4b2e8fdf6297c23395"
server_key_version: "encryptionkey@0"
I20260810 06:20:21.620080 4567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "da76d2a3616641058aa844a02b66bfdc"
format_stamp: "Formatted at 2026-08-10 06:20:21 on dist-test-slave-q9h9"
server_key: "4af1a148e6f1efe484e053b549ec83c3"
server_key_iv: "18fcf41594691b4b2e8fdf6297c23395"
server_key_version: "encryptionkey@0"
I20260810 06:20:21.623442 4567 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260810 06:20:21.625654 4589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:21.627036 4567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 06:20:21.627143 4567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "da76d2a3616641058aa844a02b66bfdc"
format_stamp: "Formatted at 2026-08-10 06:20:21 on dist-test-slave-q9h9"
server_key: "4af1a148e6f1efe484e053b549ec83c3"
server_key_iv: "18fcf41594691b4b2e8fdf6297c23395"
server_key_version: "encryptionkey@0"
I20260810 06:20:21.627245 4567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:21.651365 4567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:21.654080 4567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:21.654232 4567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:21.654733 4567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:21.655471 4567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:21.655515 4567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:21.655552 4567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:21.655582 4567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:21.663676 4567 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:41591
I20260810 06:20:21.663702 4802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:41591 every 8 connection(s)
I20260810 06:20:21.664424 4567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:20:21.669337 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 4567
I20260810 06:20:21.669437 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:20:21.669677 32563 external_mini_cluster.cc:1428] Setting key 60db8b62ccdbc5ceaeca799f63c6a9e9
Aug 10 06:20:21 dist-test-slave-q9h9 krb5kdc[3974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342821, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:21.676487 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:21.665970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:42703 (local address 127.31.204.254:39105)
0810 06:20:21.666120 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:21.666124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:21.666811 (+ 687us) server_negotiation.cc:407] Connection header received
0810 06:20:21.668021 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:21.668024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:21.668079 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:21.668153 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:21.669179 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:21.669752 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:21.671054 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:21.671284 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:21.671368 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:21.673565 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:21.673569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:21.673571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:21.673596 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:21.674698 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:21.675148 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:21.675151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:21.675153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:21.675231 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:21.675527 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:21.675531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:21.675533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:21.675703 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:21.675794 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:21.676217 (+ 423us) server_negotiation.cc:299] Negotiation successful
0810 06:20:21.676336 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 06:20:21.677117 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:21.666295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:39105 (local address 127.31.204.195:42703)
0810 06:20:21.666686 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:21.666723 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:21.667832 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:21.668318 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:21.668324 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:21.668625 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:21.669028 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:21.669040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:21.669873 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:21.669875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:21.670949 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:21.670960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:21.671422 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:21.671426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:21.671595 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:21.672108 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:21.672123 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:21.673415 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:21.674799 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:21.674815 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:21.674827 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:21.675056 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:21.675314 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:21.675317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:21.675319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:21.675442 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:21.675798 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:21.675803 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:21.676008 (+ 205us) client_negotiation.cc:769] Sending connection context
0810 06:20:21.676158 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 06:20:21.676441 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":110,"threads_started":1}
I20260810 06:20:21.678028 4803 heartbeater.cc:347] Connected to a master server at 127.31.204.254:39105
I20260810 06:20:21.678232 4803 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:21.678639 4803 heartbeater.cc:511] Master 127.31.204.254:39105 requested a full tablet report, sending...
I20260810 06:20:21.679493 4045 ts_manager.cc:194] Registered new tserver with Master: da76d2a3616641058aa844a02b66bfdc (127.31.204.195:41591)
I20260810 06:20:21.680006 4045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:42703
I20260810 06:20:21.683003 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 06:20:21.696426 4100 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:21.695179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55906 (local address 127.31.204.254:39105)
0810 06:20:21.695278 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:21.695282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:21.695494 (+ 212us) server_negotiation.cc:407] Connection header received
0810 06:20:21.695606 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:21.695609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:21.695646 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:21.695706 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:21.696339 (+ 633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55906: BlockingRecv error: recv got EOF from 127.0.0.1:55906 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 06:20:21.697365 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 4140
I20260810 06:20:21.723326 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 4357
I20260810 06:20:21.735673 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 4567
I20260810 06:20:21.744799 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 3996
2026-08-10T06:20:21Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3153 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:20:21 dist-test-slave-q9h9 krb5kdc[4819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:20:21 dist-test-slave-q9h9 krb5kdc[4819](info): set up 2 sockets
Aug 10 06:20:21 dist-test-slave-q9h9 krb5kdc[4819](info): commencing operation
krb5kdc: starting...
W20260810 06:20:22.900274 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:20:22 dist-test-slave-q9h9 krb5kdc[4819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:20:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:20:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:20:23.011027 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:38761
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:38761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:23.116966 4835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:23.117218 4835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:23.117275 4835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:23.121277 4835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:23.121335 4835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:23.121362 4835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:23.121383 4835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:23.121410 4835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:20:23.126467 4835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:38761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:38761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:23.127431 4835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:23.128401 4835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:23.136354 4845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:23.136339 4842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:23.136332 4841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:23.136588 4835 server_base.cc:1034] running on GCE node
I20260810 06:20:23.137161 4835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:23.138216 4835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:23.139406 4835 hybrid_clock.cc:648] HybridClock initialized: now 1786342823139392 us; error 31 us; skew 500 ppm
Aug 10 06:20:23 dist-test-slave-q9h9 krb5kdc[4819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342823, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:23.141993 4835 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:20:23.142872 4835 webserver.cc:460] Webserver started at http://127.31.204.254:46651/ using document root <none> and password file <none>
I20260810 06:20:23.143384 4835 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:23.143446 4835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:23.143636 4835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:23.145556 4835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "bc9c9e1966d34a6087006981b320061a"
format_stamp: "Formatted at 2026-08-10 06:20:23 on dist-test-slave-q9h9"
server_key: "24acc19e6cfdee2d27c520c0872570aa"
server_key_iv: "495fb8ee045d134cd5500c4034f3cd7c"
server_key_version: "encryptionkey@0"
I20260810 06:20:23.146116 4835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "bc9c9e1966d34a6087006981b320061a"
format_stamp: "Formatted at 2026-08-10 06:20:23 on dist-test-slave-q9h9"
server_key: "24acc19e6cfdee2d27c520c0872570aa"
server_key_iv: "495fb8ee045d134cd5500c4034f3cd7c"
server_key_version: "encryptionkey@0"
I20260810 06:20:23.149542 4835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:20:23.151827 4856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:23.152936 4835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:23.153036 4835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "bc9c9e1966d34a6087006981b320061a"
format_stamp: "Formatted at 2026-08-10 06:20:23 on dist-test-slave-q9h9"
server_key: "24acc19e6cfdee2d27c520c0872570aa"
server_key_iv: "495fb8ee045d134cd5500c4034f3cd7c"
server_key_version: "encryptionkey@0"
I20260810 06:20:23.153129 4835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:23.162218 4835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:23.164637 4835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:23.164794 4835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:23.171504 4835 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:38761
I20260810 06:20:23.171756 4958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:38761 every 8 connection(s)
I20260810 06:20:23.172375 4835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:20:23.174957 4959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:23.175850 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 4835
I20260810 06:20:23.175949 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:20:23.176201 32563 external_mini_cluster.cc:1428] Setting key 0e86ebb446d7c4070def0aeaad0f5a80
I20260810 06:20:23.180684 4959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a: Bootstrap starting.
I20260810 06:20:23.183171 4959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:23.184047 4959 log.cc:824] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a: Log is configured to *not* fsync() on all Append() calls
Aug 10 06:20:23 dist-test-slave-q9h9 krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:23.186110 4959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a: No bootstrap required, opened a new log
I20260810 06:20:23.189896 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:23.177333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39284 (local address 127.31.204.254:38761)
0810 06:20:23.177711 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:23.177725 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:23.177758 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:20:23.178411 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:23.178428 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:23.178734 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:23.178987 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:23.179872 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:23.180819 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:23.181940 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:23.182416 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:23.182529 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:23.185201 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:23.185207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:23.185222 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:23.185251 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:23.187118 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:23.187511 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:23.187517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:23.187522 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:23.187660 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:23.187880 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:23.187883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:23.187885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:23.188392 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:23.188573 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:23.188750 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 06:20:23.189024 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":100,"threads_started":1}
I20260810 06:20:23.191119 4959 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9c9e1966d34a6087006981b320061a" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38761 } }
I20260810 06:20:23.191337 4959 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:23.191394 4959 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc9c9e1966d34a6087006981b320061a, State: Initialized, Role: FOLLOWER
I20260810 06:20:23.191847 4959 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9c9e1966d34a6087006981b320061a" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38761 } }
I20260810 06:20:23.192003 4959 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:20:23.192070 4959 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:20:23.192163 4959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:23.192394 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:20:23.193015 4959 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9c9e1966d34a6087006981b320061a" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38761 } }
I20260810 06:20:23.193308 4959 leader_election.cc:302] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc9c9e1966d34a6087006981b320061a; no voters:
I20260810 06:20:23.193563 4959 leader_election.cc:288] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:20:23.193665 4964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:23.193849 4964 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a [term 1 LEADER]: Becoming Leader. State: Replica: bc9c9e1966d34a6087006981b320061a, State: Running, Role: LEADER
I20260810 06:20:23.194203 4964 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9c9e1966d34a6087006981b320061a" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38761 } }
I20260810 06:20:23.194764 4959 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:23.195691 4968 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc9c9e1966d34a6087006981b320061a. Latest consensus state: current_term: 1 leader_uuid: "bc9c9e1966d34a6087006981b320061a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9c9e1966d34a6087006981b320061a" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38761 } } }
I20260810 06:20:23.195809 4968 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:23.196197 4970 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc9c9e1966d34a6087006981b320061a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9c9e1966d34a6087006981b320061a" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38761 } } }
I20260810 06:20:23.196305 4970 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:23.196951 4984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:23.199739 4984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:23.205421 4984 catalog_manager.cc:1347] Generated new cluster ID: c9dbeef6fc3e4c74a29b911b70d9da97
I20260810 06:20:23.205482 4984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:23.210404 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:23.206105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39298 (local address 127.31.204.254:38761)
0810 06:20:23.206352 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:23.206356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:23.206576 (+ 220us) server_negotiation.cc:407] Connection header received
0810 06:20:23.206701 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:23.206703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:23.206741 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:23.206799 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:23.207268 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:23.207759 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:23.208547 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:23.208788 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:23.208873 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:23.209580 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:23.209584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:23.209585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:20:23.209960 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:23.209985 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:20:23.210000 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:23.210097 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:23.210232 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 06:20:23.210284 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260810 06:20:23.214684 4984 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:20:23.215739 4984 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:23.226089 4984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc9c9e1966d34a6087006981b320061a: Generated new TSK 0
I20260810 06:20:23.226668 4984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:20:23.235280 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 4835
2026-08-10T06:20:23Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1488 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:20:23 dist-test-slave-q9h9 krb5kdc[5015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:20:23 dist-test-slave-q9h9 krb5kdc[5015](info): set up 2 sockets
Aug 10 06:20:23 dist-test-slave-q9h9 krb5kdc[5015](info): commencing operation
krb5kdc: starting...
W20260810 06:20:24.408589 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:20:24 dist-test-slave-q9h9 krb5kdc[5015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:20:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:20:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:20:24.517992 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36235
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:36235
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:24.622392 5035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:24.622618 5035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:24.622671 5035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:24.626582 5035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:24.626631 5035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:24.626658 5035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:24.626680 5035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:24.626709 5035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:20:24.631632 5035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:36235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:24.632500 5035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:24.633406 5035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:24.641075 5045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:24.641100 5041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:24.641131 5042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:24.641211 5035 server_base.cc:1034] running on GCE node
I20260810 06:20:24.641955 5035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:24.642937 5035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:24.644089 5035 hybrid_clock.cc:648] HybridClock initialized: now 1786342824644073 us; error 29 us; skew 500 ppm
Aug 10 06:20:24 dist-test-slave-q9h9 krb5kdc[5015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342824, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:24.651089 5035 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:20:24.651891 5035 webserver.cc:460] Webserver started at http://127.31.204.254:45909/ using document root <none> and password file <none>
I20260810 06:20:24.652371 5035 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:24.652436 5035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:24.652626 5035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:24.654541 5035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "14205dc769fc4fac93a2fc7972e0176b"
format_stamp: "Formatted at 2026-08-10 06:20:24 on dist-test-slave-q9h9"
server_key: "ac0b456b04048a8f45e1a37d58cb3c0a"
server_key_iv: "e2b88bbb7ad38a6a94979944b127887e"
server_key_version: "encryptionkey@0"
I20260810 06:20:24.655117 5035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "14205dc769fc4fac93a2fc7972e0176b"
format_stamp: "Formatted at 2026-08-10 06:20:24 on dist-test-slave-q9h9"
server_key: "ac0b456b04048a8f45e1a37d58cb3c0a"
server_key_iv: "e2b88bbb7ad38a6a94979944b127887e"
server_key_version: "encryptionkey@0"
I20260810 06:20:24.658469 5035 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:20:24.660638 5055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:24.661728 5035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:24.661845 5035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "14205dc769fc4fac93a2fc7972e0176b"
format_stamp: "Formatted at 2026-08-10 06:20:24 on dist-test-slave-q9h9"
server_key: "ac0b456b04048a8f45e1a37d58cb3c0a"
server_key_iv: "e2b88bbb7ad38a6a94979944b127887e"
server_key_version: "encryptionkey@0"
I20260810 06:20:24.661929 5035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:24.678221 5035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:24.680933 5035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:24.681092 5035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:24.687783 5035 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:36235
I20260810 06:20:24.687806 5145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:36235 every 8 connection(s)
I20260810 06:20:24.689252 5035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:20:24.691867 5146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:24.692840 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 5035
I20260810 06:20:24.692941 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:20:24.693190 32563 external_mini_cluster.cc:1428] Setting key 86216f412e2ea0a56fcb895772e11620
I20260810 06:20:24.698185 5146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b: Bootstrap starting.
I20260810 06:20:24.700637 5146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:24.701324 5146 log.cc:824] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b: Log is configured to *not* fsync() on all Append() calls
Aug 10 06:20:24 dist-test-slave-q9h9 krb5kdc[5015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:24.703289 5146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b: No bootstrap required, opened a new log
I20260810 06:20:24.706527 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:24.694430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.31.204.254:36235)
0810 06:20:24.694854 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:24.694874 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:24.694918 (+ 44us) server_negotiation.cc:407] Connection header received
0810 06:20:24.695705 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:24.695727 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:24.696111 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:24.696454 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:24.697079 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:24.698519 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:24.699328 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:24.700005 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:24.700150 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:24.702481 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:24.702485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:24.702497 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:24.702522 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:24.704044 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:24.704491 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:24.704495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:24.704499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:24.704601 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:24.704854 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:24.704856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:24.704858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:24.705202 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:24.705334 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:24.705589 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 06:20:24.705839 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":112,"threads_started":1}
I20260810 06:20:24.706938 5146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14205dc769fc4fac93a2fc7972e0176b" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36235 } }
I20260810 06:20:24.707130 5146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:24.707181 5146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14205dc769fc4fac93a2fc7972e0176b, State: Initialized, Role: FOLLOWER
I20260810 06:20:24.707659 5146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14205dc769fc4fac93a2fc7972e0176b" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36235 } }
I20260810 06:20:24.707813 5146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:20:24.707877 5146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:20:24.707967 5146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:24.708814 5146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14205dc769fc4fac93a2fc7972e0176b" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36235 } }
I20260810 06:20:24.709080 5146 leader_election.cc:302] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14205dc769fc4fac93a2fc7972e0176b; no voters:
I20260810 06:20:24.709323 5146 leader_election.cc:288] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:20:24.709412 5154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:24.709610 5154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b [term 1 LEADER]: Becoming Leader. State: Replica: 14205dc769fc4fac93a2fc7972e0176b, State: Running, Role: LEADER
I20260810 06:20:24.709939 5154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14205dc769fc4fac93a2fc7972e0176b" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36235 } }
I20260810 06:20:24.710384 5146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:24.711563 5159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14205dc769fc4fac93a2fc7972e0176b. Latest consensus state: current_term: 1 leader_uuid: "14205dc769fc4fac93a2fc7972e0176b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14205dc769fc4fac93a2fc7972e0176b" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36235 } } }
I20260810 06:20:24.711572 5156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14205dc769fc4fac93a2fc7972e0176b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14205dc769fc4fac93a2fc7972e0176b" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36235 } } }
I20260810 06:20:24.711687 5159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:24.711701 5156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:24.712062 5174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:24.714843 5174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:24.719904 5174 catalog_manager.cc:1347] Generated new cluster ID: 96c5827791a749be8c631e47a083bd51
I20260810 06:20:24.719962 5174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:24.734050 5174 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:20:24.735247 5174 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:24.741477 5174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14205dc769fc4fac93a2fc7972e0176b: Generated new TSK 0
I20260810 06:20:24.742166 5174 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
I20260810 06:20:24.778321 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36235
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:24.885263 5192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:24.885484 5192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:24.885540 5192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:24.889387 5192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:24.889437 5192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:24.889514 5192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:20:24.894677 5192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:24.895598 5192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:24.896526 5192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:24.905329 5206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:24.905337 5210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:24.905337 5207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:24.905553 5192 server_base.cc:1034] running on GCE node
I20260810 06:20:24.905925 5192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:24.906538 5192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:24.907706 5192 hybrid_clock.cc:648] HybridClock initialized: now 1786342824907689 us; error 30 us; skew 500 ppm
Aug 10 06:20:24 dist-test-slave-q9h9 krb5kdc[5015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342824, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:24.910257 5192 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:20:24.911150 5192 webserver.cc:460] Webserver started at http://127.31.204.193:42445/ using document root <none> and password file <none>
I20260810 06:20:24.911682 5192 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:24.911756 5192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:24.911954 5192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:24.913872 5192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "a1ed2e6fa87d4b44a3193f8bd3d4f5e0"
format_stamp: "Formatted at 2026-08-10 06:20:24 on dist-test-slave-q9h9"
server_key: "67cb5bc233a523b7698978d384fd89a6"
server_key_iv: "1f8ff7625cb79d62922f5983500b0bca"
server_key_version: "encryptionkey@0"
I20260810 06:20:24.914449 5192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "a1ed2e6fa87d4b44a3193f8bd3d4f5e0"
format_stamp: "Formatted at 2026-08-10 06:20:24 on dist-test-slave-q9h9"
server_key: "67cb5bc233a523b7698978d384fd89a6"
server_key_iv: "1f8ff7625cb79d62922f5983500b0bca"
server_key_version: "encryptionkey@0"
I20260810 06:20:24.917856 5192 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 06:20:24.920110 5219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:24.921203 5192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:24.921308 5192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "a1ed2e6fa87d4b44a3193f8bd3d4f5e0"
format_stamp: "Formatted at 2026-08-10 06:20:24 on dist-test-slave-q9h9"
server_key: "67cb5bc233a523b7698978d384fd89a6"
server_key_iv: "1f8ff7625cb79d62922f5983500b0bca"
server_key_version: "encryptionkey@0"
I20260810 06:20:24.921394 5192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:24.938611 5192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:24.940992 5192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:24.941151 5192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:24.941672 5192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:24.942468 5192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:24.942514 5192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:24.942564 5192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:24.942612 5192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:24.951355 5192 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:44075
I20260810 06:20:24.951380 5409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:44075 every 8 connection(s)
I20260810 06:20:24.952171 5192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:20:24.953896 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 5192
I20260810 06:20:24.954041 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:20:24.954363 32563 external_mini_cluster.cc:1428] Setting key 4de171e8198f099d43a352f9aed7a38c
Aug 10 06:20:24 dist-test-slave-q9h9 krb5kdc[5015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342824, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.204.194@KRBTEST.COM; defaulting to no policy
I20260810 06:20:24.965909 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:24.953749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:38097 (local address 127.31.204.254:36235)
0810 06:20:24.953872 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:24.953875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:24.954860 (+ 985us) server_negotiation.cc:407] Connection header received
0810 06:20:24.956234 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:24.956237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:24.956291 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:24.956366 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:24.957461 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:24.957989 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:24.959508 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:24.959732 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:24.959817 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:24.962610 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:24.962613 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:24.962615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:24.962639 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:24.963893 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:24.964453 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:24.964456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:24.964458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:24.964537 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:24.964890 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:24.964892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:24.964893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:24.965057 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:24.965137 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:24.965622 (+ 485us) server_negotiation.cc:299] Negotiation successful
0810 06:20:24.965744 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.194@KRBTEST.COM" created.
I20260810 06:20:24.966744 5413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:24.954226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36235 (local address 127.31.204.193:38097)
0810 06:20:24.954708 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:24.954751 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:24.956035 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:24.956532 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:24.956538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:24.956883 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:24.957321 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:24.957334 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:24.958135 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:24.958139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:24.959394 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:24.959407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:24.959871 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:24.959875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:24.960070 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:24.960702 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:24.960717 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:24.962438 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:24.964011 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:24.964017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:24.964030 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:24.964344 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:24.964640 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:24.964644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:24.964646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:24.964819 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:24.965139 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:24.965144 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:24.965398 (+ 254us) client_negotiation.cc:769] Sending connection context
0810 06:20:24.965555 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 06:20:24.965864 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":132,"threads_started":1}
I20260810 06:20:24.967916 5410 heartbeater.cc:347] Connected to a master server at 127.31.204.254:36235
I20260810 06:20:24.968124 5410 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:24.968573 5410 heartbeater.cc:511] Master 127.31.204.254:36235 requested a full tablet report, sending...
I20260810 06:20:24.970008 5072 ts_manager.cc:194] Registered new tserver with Master: a1ed2e6fa87d4b44a3193f8bd3d4f5e0 (127.31.204.193:44075)
I20260810 06:20:24.971161 5072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:38097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
I20260810 06:20:24.981040 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36235
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:25.086539 5419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:25.086756 5419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:25.086794 5419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:25.090639 5419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:25.090691 5419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:25.090744 5419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:20:25.095662 5419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:25.096540 5419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:25.097431 5419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:25.106195 5433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:25.106197 5428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:25.106199 5431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:25.106467 5419 server_base.cc:1034] running on GCE node
I20260810 06:20:25.106901 5419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:25.107465 5419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:25.108618 5419 hybrid_clock.cc:648] HybridClock initialized: now 1786342825108600 us; error 37 us; skew 500 ppm
Aug 10 06:20:25 dist-test-slave-q9h9 krb5kdc[5015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342825, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:25.111238 5419 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:20:25.112085 5419 webserver.cc:460] Webserver started at http://127.31.204.194:42393/ using document root <none> and password file <none>
I20260810 06:20:25.112582 5419 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:25.112639 5419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:25.112820 5419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:25.114746 5419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "61c41bbedecd40758072a9b8f734c630"
format_stamp: "Formatted at 2026-08-10 06:20:25 on dist-test-slave-q9h9"
server_key: "ed2e4c5d5f3e972de2726f13d08ece3c"
server_key_iv: "4cb62495f5c8118b4b03cd55bcfe96c5"
server_key_version: "encryptionkey@0"
I20260810 06:20:25.115275 5419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "61c41bbedecd40758072a9b8f734c630"
format_stamp: "Formatted at 2026-08-10 06:20:25 on dist-test-slave-q9h9"
server_key: "ed2e4c5d5f3e972de2726f13d08ece3c"
server_key_iv: "4cb62495f5c8118b4b03cd55bcfe96c5"
server_key_version: "encryptionkey@0"
I20260810 06:20:25.118739 5419 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:20:25.120903 5446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:25.122047 5419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:25.122154 5419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "61c41bbedecd40758072a9b8f734c630"
format_stamp: "Formatted at 2026-08-10 06:20:25 on dist-test-slave-q9h9"
server_key: "ed2e4c5d5f3e972de2726f13d08ece3c"
server_key_iv: "4cb62495f5c8118b4b03cd55bcfe96c5"
server_key_version: "encryptionkey@0"
I20260810 06:20:25.122251 5419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:25.136099 5419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:25.138635 5419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:25.138772 5419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:25.139257 5419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:25.139997 5419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:25.140043 5419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:25.140079 5419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:25.140097 5419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:25.148522 5419 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:43287
I20260810 06:20:25.148543 5619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:43287 every 8 connection(s)
I20260810 06:20:25.149314 5419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:20:25.156679 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 5419
I20260810 06:20:25.156802 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:20:25.157094 32563 external_mini_cluster.cc:1428] Setting key c70466777514bd07c8584539faa4e416
Aug 10 06:20:25 dist-test-slave-q9h9 krb5kdc[5015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342825, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:25.162333 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:25.150986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:59671 (local address 127.31.204.254:36235)
0810 06:20:25.151106 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:25.151111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:25.151935 (+ 824us) server_negotiation.cc:407] Connection header received
0810 06:20:25.153263 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:25.153266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:25.153336 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:25.153408 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:25.154547 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:25.155084 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:25.156183 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:25.156411 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:25.156511 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:25.158888 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:25.158893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:25.158895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:25.158924 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:25.160429 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:25.160934 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:25.160937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:25.160938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:25.161018 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:25.161328 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:25.161332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:25.161333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:25.161505 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:25.161591 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:25.162097 (+ 506us) server_negotiation.cc:299] Negotiation successful
0810 06:20:25.162200 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 06:20:25.163235 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:25.151312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36235 (local address 127.31.204.194:59671)
0810 06:20:25.151774 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:25.151822 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:25.153064 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:25.153600 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:25.153606 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:25.153936 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:25.154411 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:25.154424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:25.155210 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:25.155213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:25.156065 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:25.156075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:25.156559 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:25.156562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:25.156701 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:25.157294 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:25.157310 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:25.158729 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:25.160538 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:25.160543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:25.160555 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:25.160836 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:25.161100 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:25.161103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:25.161106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:25.161241 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:25.161604 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:25.161611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:25.161817 (+ 206us) client_negotiation.cc:769] Sending connection context
0810 06:20:25.162006 (+ 189us) client_negotiation.cc:240] Negotiation successful
0810 06:20:25.162344 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260810 06:20:25.164263 5620 heartbeater.cc:347] Connected to a master server at 127.31.204.254:36235
I20260810 06:20:25.164481 5620 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:25.164945 5620 heartbeater.cc:511] Master 127.31.204.254:36235 requested a full tablet report, sending...
I20260810 06:20:25.165894 5072 ts_manager.cc:194] Registered new tserver with Master: 61c41bbedecd40758072a9b8f734c630 (127.31.204.194:43287)
I20260810 06:20:25.166514 5072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:59671
WARNING: no policy specified for kudu/127.31.204.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
I20260810 06:20:25.182498 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36235
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:25.288842 5630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:25.289057 5630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:25.289095 5630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:25.292894 5630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:25.292941 5630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:25.292994 5630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:20:25.297832 5630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:25.298694 5630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:25.299561 5630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:25.308377 5643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:25.308355 5645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:25.308364 5641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:25.308697 5630 server_base.cc:1034] running on GCE node
I20260810 06:20:25.309038 5630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:25.309553 5630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:25.310694 5630 hybrid_clock.cc:648] HybridClock initialized: now 1786342825310685 us; error 27 us; skew 500 ppm
Aug 10 06:20:25 dist-test-slave-q9h9 krb5kdc[5015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342825, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:25.313265 5630 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:20:25.313999 5630 webserver.cc:460] Webserver started at http://127.31.204.195:38025/ using document root <none> and password file <none>
I20260810 06:20:25.314507 5630 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:25.314565 5630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:25.314744 5630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:25.316633 5630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "4fc1e9047f954efc8e9b5b376acdab0b"
format_stamp: "Formatted at 2026-08-10 06:20:25 on dist-test-slave-q9h9"
server_key: "53ad1323138cba2bb566da6f3daa7501"
server_key_iv: "85a4627383a1d4a5c2d92c24295440e3"
server_key_version: "encryptionkey@0"
I20260810 06:20:25.317168 5630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "4fc1e9047f954efc8e9b5b376acdab0b"
format_stamp: "Formatted at 2026-08-10 06:20:25 on dist-test-slave-q9h9"
server_key: "53ad1323138cba2bb566da6f3daa7501"
server_key_iv: "85a4627383a1d4a5c2d92c24295440e3"
server_key_version: "encryptionkey@0"
I20260810 06:20:25.320482 5630 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260810 06:20:25.322670 5657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:25.323726 5630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 06:20:25.323827 5630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "4fc1e9047f954efc8e9b5b376acdab0b"
format_stamp: "Formatted at 2026-08-10 06:20:25 on dist-test-slave-q9h9"
server_key: "53ad1323138cba2bb566da6f3daa7501"
server_key_iv: "85a4627383a1d4a5c2d92c24295440e3"
server_key_version: "encryptionkey@0"
I20260810 06:20:25.323920 5630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:25.356410 5630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:25.358901 5630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:25.359050 5630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:25.359545 5630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:25.360289 5630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:25.360334 5630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:25.360374 5630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:25.360402 5630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:25.368554 5630 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:38097
I20260810 06:20:25.368594 5835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:38097 every 8 connection(s)
I20260810 06:20:25.369315 5630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
Aug 10 06:20:25 dist-test-slave-q9h9 krb5kdc[5015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342825, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:25.377669 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 5630
I20260810 06:20:25.377774 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:20:25.378039 32563 external_mini_cluster.cc:1428] Setting key 7987390939a690019f4cf04517805f2b
I20260810 06:20:25.381184 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:25.370890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:35579 (local address 127.31.204.254:36235)
0810 06:20:25.371024 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:25.371028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:25.371789 (+ 761us) server_negotiation.cc:407] Connection header received
0810 06:20:25.373010 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:25.373013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:25.373057 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:25.373125 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:25.374164 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:25.374671 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:25.375715 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:25.375916 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:25.375982 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:25.378123 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:25.378127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:25.378130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:25.378153 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:25.379550 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:25.380090 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:25.380093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:25.380094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:25.380166 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:25.380414 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:25.380416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:25.380417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:25.380563 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:25.380629 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:25.380985 (+ 356us) server_negotiation.cc:299] Negotiation successful
0810 06:20:25.381072 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:20:25.381929 5840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:25.371231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36235 (local address 127.31.204.195:35579)
0810 06:20:25.371647 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:25.371685 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:25.372841 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:25.373281 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:25.373287 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:25.373608 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:25.374046 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:25.374060 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:25.374776 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:25.374779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:25.375625 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:25.375634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:25.376013 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:25.376016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:25.376154 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:25.376575 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:25.376589 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:25.377944 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:25.379659 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:25.379664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:25.379677 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:25.380000 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:25.380232 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:25.380234 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:25.380235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:25.380352 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:25.380629 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:25.380633 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:25.380821 (+ 188us) client_negotiation.cc:769] Sending connection context
0810 06:20:25.380956 (+ 135us) client_negotiation.cc:240] Negotiation successful
0810 06:20:25.381181 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
I20260810 06:20:25.382869 5837 heartbeater.cc:347] Connected to a master server at 127.31.204.254:36235
I20260810 06:20:25.383064 5837 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:25.383495 5837 heartbeater.cc:511] Master 127.31.204.254:36235 requested a full tablet report, sending...
I20260810 06:20:25.384346 5072 ts_manager.cc:194] Registered new tserver with Master: 4fc1e9047f954efc8e9b5b376acdab0b (127.31.204.195:38097)
I20260810 06:20:25.384845 5072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:35579
I20260810 06:20:25.391208 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 06:20:25 dist-test-slave-q9h9 krb5kdc[5015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342825, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 06:20:25 dist-test-slave-q9h9 krb5kdc[5015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342825, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:25.414743 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:25.407646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55376 (local address 127.31.204.254:36235)
0810 06:20:25.407751 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:25.407755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:25.407920 (+ 165us) server_negotiation.cc:407] Connection header received
0810 06:20:25.408024 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:25.408026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:25.408059 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:25.408113 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:25.408712 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:25.409177 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:25.409902 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:25.410138 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:25.410204 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:25.412474 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:25.412477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:25.412479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:25.412500 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:25.413510 (+ 1010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:25.413843 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:25.413845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:25.413846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:25.413907 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:25.414126 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:25.414128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:25.414129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:25.414310 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:25.414405 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:25.414555 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 06:20:25.414639 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 06:20:25.415047 5072 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55376
I20260810 06:20:25.415836 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 5192
I20260810 06:20:25.437924 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 5419
I20260810 06:20:25.448518 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 5630
I20260810 06:20:25.458426 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 5035
2026-08-10T06:20:25Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2221 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:20:25 dist-test-slave-q9h9 krb5kdc[5854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:20:25 dist-test-slave-q9h9 krb5kdc[5854](info): set up 2 sockets
Aug 10 06:20:25 dist-test-slave-q9h9 krb5kdc[5854](info): commencing operation
krb5kdc: starting...
W20260810 06:20:26.615868 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:20:26 dist-test-slave-q9h9 krb5kdc[5854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:20:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:20:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:20:26.721549 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:34307
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:43399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:34307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:26.825336 5875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:26.825557 5875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:26.825610 5875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:26.829453 5875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:26.829502 5875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:26.829528 5875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:26.829548 5875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:26.829572 5875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:20:26.834506 5875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:43399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:34307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:34307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:26.835346 5875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:26.836233 5875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:26.843503 5885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:26.843564 5884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:26.843675 5888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:26.843569 5875 server_base.cc:1034] running on GCE node
I20260810 06:20:26.844242 5875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:26.845178 5875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:26.846333 5875 hybrid_clock.cc:648] HybridClock initialized: now 1786342826846312 us; error 33 us; skew 500 ppm
Aug 10 06:20:26 dist-test-slave-q9h9 krb5kdc[5854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342826, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:26.848750 5875 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:20:26.849488 5875 webserver.cc:460] Webserver started at http://127.31.204.254:34319/ using document root <none> and password file <none>
I20260810 06:20:26.849956 5875 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:26.850037 5875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:26.850229 5875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:26.852035 5875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "116f7639b11b41d8aa7e0f4de8344ace"
format_stamp: "Formatted at 2026-08-10 06:20:26 on dist-test-slave-q9h9"
server_key: "ab8dceee1961cd85b0fe2b61145ba821"
server_key_iv: "0a1f73cc9c647be18a737931270c79ae"
server_key_version: "encryptionkey@0"
I20260810 06:20:26.852643 5875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "116f7639b11b41d8aa7e0f4de8344ace"
format_stamp: "Formatted at 2026-08-10 06:20:26 on dist-test-slave-q9h9"
server_key: "ab8dceee1961cd85b0fe2b61145ba821"
server_key_iv: "0a1f73cc9c647be18a737931270c79ae"
server_key_version: "encryptionkey@0"
I20260810 06:20:26.856026 5875 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 06:20:26.858079 5897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:26.859131 5875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:26.859236 5875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "116f7639b11b41d8aa7e0f4de8344ace"
format_stamp: "Formatted at 2026-08-10 06:20:26 on dist-test-slave-q9h9"
server_key: "ab8dceee1961cd85b0fe2b61145ba821"
server_key_iv: "0a1f73cc9c647be18a737931270c79ae"
server_key_version: "encryptionkey@0"
I20260810 06:20:26.859319 5875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:26.874823 5875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:26.877136 5875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:26.877291 5875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:26.883509 5875 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:34307
I20260810 06:20:26.883556 5987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:34307 every 8 connection(s)
I20260810 06:20:26.884312 5875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:20:26.886124 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 5875
I20260810 06:20:26.886240 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:20:26.886487 32563 external_mini_cluster.cc:1428] Setting key 81a7e4c4334be7af9ad4014b3e71820b
I20260810 06:20:26.886869 5988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:26.893270 5988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace: Bootstrap starting.
Aug 10 06:20:26 dist-test-slave-q9h9 krb5kdc[5854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:26.895679 5988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:26.896461 5988 log.cc:824] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:26.898403 5988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace: No bootstrap required, opened a new log
I20260810 06:20:26.899735 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:26.887714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35174 (local address 127.31.204.254:34307)
0810 06:20:26.888123 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:26.888142 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:26.888181 (+ 39us) server_negotiation.cc:407] Connection header received
0810 06:20:26.888876 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:26.888897 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:26.889284 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:26.889620 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:26.890297 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:26.891686 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:26.892454 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:26.893095 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:26.893256 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:26.895661 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:26.895665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:26.895677 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:26.895709 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:26.897174 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:26.897649 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:26.897654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:26.897658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:26.897769 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:26.898078 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:26.898081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:26.898082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:26.898414 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:26.898593 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:26.898840 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 06:20:26.899083 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":121,"threads_started":1}
I20260810 06:20:26.901722 5988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116f7639b11b41d8aa7e0f4de8344ace" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34307 } }
I20260810 06:20:26.901908 5988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:26.901966 5988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116f7639b11b41d8aa7e0f4de8344ace, State: Initialized, Role: FOLLOWER
I20260810 06:20:26.902338 5988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116f7639b11b41d8aa7e0f4de8344ace" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34307 } }
I20260810 06:20:26.902451 5988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:20:26.902508 5988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:20:26.902591 5988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:26.903322 5988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116f7639b11b41d8aa7e0f4de8344ace" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34307 } }
I20260810 06:20:26.903605 5988 leader_election.cc:302] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 116f7639b11b41d8aa7e0f4de8344ace; no voters:
I20260810 06:20:26.903890 5988 leader_election.cc:288] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:20:26.903960 5998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:26.904181 5998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace [term 1 LEADER]: Becoming Leader. State: Replica: 116f7639b11b41d8aa7e0f4de8344ace, State: Running, Role: LEADER
I20260810 06:20:26.904501 5998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116f7639b11b41d8aa7e0f4de8344ace" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34307 } }
I20260810 06:20:26.904999 5988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:26.906065 6001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "116f7639b11b41d8aa7e0f4de8344ace" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116f7639b11b41d8aa7e0f4de8344ace" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34307 } } }
I20260810 06:20:26.906219 6001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:26.906065 6007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace [sys.catalog]: SysCatalogTable state changed. Reason: New leader 116f7639b11b41d8aa7e0f4de8344ace. Latest consensus state: current_term: 1 leader_uuid: "116f7639b11b41d8aa7e0f4de8344ace" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116f7639b11b41d8aa7e0f4de8344ace" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34307 } } }
I20260810 06:20:26.906571 6007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:26.906602 6015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:26.909504 6015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:26.916994 6015 catalog_manager.cc:1347] Generated new cluster ID: 8eb9051122294a64afdb5ecdf21666bd
I20260810 06:20:26.917045 6015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:26.937041 6015 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:20:26.937819 6015 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:26.943116 6015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 116f7639b11b41d8aa7e0f4de8344ace: Generated new TSK 0
I20260810 06:20:26.943650 6015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
I20260810 06:20:26.974740 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34307
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:43399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:27.081323 6048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:27.081533 6048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:27.081576 6048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:27.085418 6048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:27.085465 6048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:27.085533 6048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:20:27.090485 6048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:43399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:27.091356 6048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:27.092258 6048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:27.100991 6057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:27.101004 6059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:27.101004 6062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:27.101053 6048 server_base.cc:1034] running on GCE node
I20260810 06:20:27.101599 6048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:27.102213 6048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:27.103366 6048 hybrid_clock.cc:648] HybridClock initialized: now 1786342827103342 us; error 35 us; skew 500 ppm
Aug 10 06:20:27 dist-test-slave-q9h9 krb5kdc[5854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342827, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:27.105921 6048 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:20:27.106770 6048 webserver.cc:460] Webserver started at http://127.31.204.193:46725/ using document root <none> and password file <none>
I20260810 06:20:27.107257 6048 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:27.107318 6048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:27.107479 6048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:27.109303 6048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "0c29a9ba045d4ec5b655662298d993c4"
format_stamp: "Formatted at 2026-08-10 06:20:27 on dist-test-slave-q9h9"
server_key: "531ca380b81297e2992dd5df6585663e"
server_key_iv: "71ac301c9ed4d82400ca84062b69e168"
server_key_version: "encryptionkey@0"
I20260810 06:20:27.109895 6048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "0c29a9ba045d4ec5b655662298d993c4"
format_stamp: "Formatted at 2026-08-10 06:20:27 on dist-test-slave-q9h9"
server_key: "531ca380b81297e2992dd5df6585663e"
server_key_iv: "71ac301c9ed4d82400ca84062b69e168"
server_key_version: "encryptionkey@0"
I20260810 06:20:27.113224 6048 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 06:20:27.115374 6074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:27.116421 6048 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 06:20:27.116513 6048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "0c29a9ba045d4ec5b655662298d993c4"
format_stamp: "Formatted at 2026-08-10 06:20:27 on dist-test-slave-q9h9"
server_key: "531ca380b81297e2992dd5df6585663e"
server_key_iv: "71ac301c9ed4d82400ca84062b69e168"
server_key_version: "encryptionkey@0"
I20260810 06:20:27.116588 6048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:27.130823 6048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:27.133580 6048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:27.133726 6048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:27.134244 6048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:27.134976 6048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:27.135023 6048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:27.135061 6048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:27.135073 6048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:27.143443 6048 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:40673
I20260810 06:20:27.143476 6262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:40673 every 8 connection(s)
I20260810 06:20:27.144239 6048 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:20:27.150110 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 6048
I20260810 06:20:27.150214 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:20:27.150472 32563 external_mini_cluster.cc:1428] Setting key 793689aa9238bdc8b307fff54faf4c14
Aug 10 06:20:27 dist-test-slave-q9h9 krb5kdc[5854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342827, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:27.157003 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:27.145823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:40821 (local address 127.31.204.254:34307)
0810 06:20:27.145940 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:27.145943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:27.146759 (+ 816us) server_negotiation.cc:407] Connection header received
0810 06:20:27.147996 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:27.147999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:27.148046 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:27.148117 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:27.149142 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:27.149658 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:27.150855 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:27.151160 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:27.151259 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:27.153667 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:27.153671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:27.153674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:27.153700 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:27.155075 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:27.155556 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:27.155559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:27.155560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:27.155635 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:27.155943 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:27.155946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:27.155948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:27.156132 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:27.156219 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:27.156710 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 06:20:27.156839 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:20:27.157719 6266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:27.146165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34307 (local address 127.31.204.193:40821)
0810 06:20:27.146599 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:27.146641 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:27.147820 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:27.148266 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:27.148271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:27.148587 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:27.149031 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:27.149044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:27.149747 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:27.149749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:27.150735 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:27.150745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:27.151322 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:27.151326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:27.151471 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:27.151991 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:27.152006 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:27.153506 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:27.155202 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:27.155208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:27.155222 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:27.155461 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:27.155718 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:27.155721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:27.155723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:27.155856 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:27.156233 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:27.156240 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:27.156461 (+ 221us) client_negotiation.cc:769] Sending connection context
0810 06:20:27.156652 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 06:20:27.156947 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":102,"threads_started":1}
I20260810 06:20:27.158838 6263 heartbeater.cc:347] Connected to a master server at 127.31.204.254:34307
I20260810 06:20:27.159075 6263 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:27.159623 6263 heartbeater.cc:511] Master 127.31.204.254:34307 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.204.194@KRBTEST.COM; defaulting to no policy
I20260810 06:20:27.161118 5933 ts_manager.cc:194] Registered new tserver with Master: 0c29a9ba045d4ec5b655662298d993c4 (127.31.204.193:40673)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.194@KRBTEST.COM" created.
I20260810 06:20:27.162389 5933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:40821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
I20260810 06:20:27.175028 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34307
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:43399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:27.279485 6274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:27.279724 6274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:27.279765 6274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:27.283649 6274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:27.283697 6274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:27.283751 6274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:20:27.288652 6274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:43399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:27.289546 6274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:27.290493 6274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:27.299500 6285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:27.299496 6289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:27.299492 6284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:27.299697 6274 server_base.cc:1034] running on GCE node
I20260810 06:20:27.300077 6274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:27.300603 6274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:27.301746 6274 hybrid_clock.cc:648] HybridClock initialized: now 1786342827301725 us; error 36 us; skew 500 ppm
Aug 10 06:20:27 dist-test-slave-q9h9 krb5kdc[5854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342827, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:27.304319 6274 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:20:27.305197 6274 webserver.cc:460] Webserver started at http://127.31.204.194:33181/ using document root <none> and password file <none>
I20260810 06:20:27.305763 6274 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:27.305828 6274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:27.306046 6274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:27.307937 6274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "7140fb0f4bfc4825a467d443007f044e"
format_stamp: "Formatted at 2026-08-10 06:20:27 on dist-test-slave-q9h9"
server_key: "b29ad821e02b20b588c877a9e818e67b"
server_key_iv: "92686ea24a3f3f6e330c35cd7f77aefd"
server_key_version: "encryptionkey@0"
I20260810 06:20:27.308446 6274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "7140fb0f4bfc4825a467d443007f044e"
format_stamp: "Formatted at 2026-08-10 06:20:27 on dist-test-slave-q9h9"
server_key: "b29ad821e02b20b588c877a9e818e67b"
server_key_iv: "92686ea24a3f3f6e330c35cd7f77aefd"
server_key_version: "encryptionkey@0"
I20260810 06:20:27.311935 6274 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:20:27.314158 6300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:27.315268 6274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:27.315363 6274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "7140fb0f4bfc4825a467d443007f044e"
format_stamp: "Formatted at 2026-08-10 06:20:27 on dist-test-slave-q9h9"
server_key: "b29ad821e02b20b588c877a9e818e67b"
server_key_iv: "92686ea24a3f3f6e330c35cd7f77aefd"
server_key_version: "encryptionkey@0"
I20260810 06:20:27.315459 6274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:27.341629 6274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:27.344156 6274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:27.344305 6274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:27.344821 6274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:27.345539 6274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:27.345583 6274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:27.345623 6274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:27.345652 6274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:27.353724 6274 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:38669
I20260810 06:20:27.353761 6476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:38669 every 8 connection(s)
I20260810 06:20:27.354496 6274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:20:27.360311 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 6274
I20260810 06:20:27.360414 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:20:27.360651 32563 external_mini_cluster.cc:1428] Setting key 98b0f20bca010a9fa2e25d83c232cc51
Aug 10 06:20:27 dist-test-slave-q9h9 krb5kdc[5854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342827, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:27.367197 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:27.356125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:35315 (local address 127.31.204.254:34307)
0810 06:20:27.356267 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:27.356270 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:27.357004 (+ 734us) server_negotiation.cc:407] Connection header received
0810 06:20:27.358345 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:27.358349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:27.358406 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:27.358477 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:27.359520 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:27.360037 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:27.361240 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:27.361524 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:27.361626 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:27.364062 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:27.364064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:27.364066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:27.364090 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:27.365322 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:27.365781 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:27.365784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:27.365785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:27.365862 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:27.366206 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:27.366209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:27.366211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:27.366381 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:27.366467 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:27.366921 (+ 454us) server_negotiation.cc:299] Negotiation successful
0810 06:20:27.367042 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:20:27.367862 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:27.356472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34307 (local address 127.31.204.194:35315)
0810 06:20:27.356868 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:27.356907 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:27.358140 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:27.358643 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:27.358649 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:27.358957 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:27.359377 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:27.359389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:27.360155 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:27.360158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:27.361127 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:27.361138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:27.361660 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:27.361663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:27.361856 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:27.362402 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:27.362421 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:27.363914 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:27.365429 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:27.365436 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:27.365448 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:27.365688 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:27.365944 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:27.365947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:27.365949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:27.366120 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:27.366480 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:27.366487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:27.366696 (+ 209us) client_negotiation.cc:769] Sending connection context
0810 06:20:27.366840 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 06:20:27.367120 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":101,"threads_started":1}
I20260810 06:20:27.368835 6478 heartbeater.cc:347] Connected to a master server at 127.31.204.254:34307
I20260810 06:20:27.369028 6478 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:27.369445 6478 heartbeater.cc:511] Master 127.31.204.254:34307 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.204.195@KRBTEST.COM; defaulting to no policy
I20260810 06:20:27.370345 5933 ts_manager.cc:194] Registered new tserver with Master: 7140fb0f4bfc4825a467d443007f044e (127.31.204.194:38669)
I20260810 06:20:27.370913 5933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:35315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
I20260810 06:20:27.384403 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34307
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:43399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:27.487752 6483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:27.487951 6483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:27.487988 6483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:27.491842 6483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:27.491887 6483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:27.491940 6483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:20:27.496740 6483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:43399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:27.497489 6483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:27.498328 6483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:27.506619 6492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:27.506776 6483 server_base.cc:1034] running on GCE node
W20260810 06:20:27.506606 6491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:27.506593 6494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:27.507287 6483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:27.507788 6483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:27.508934 6483 hybrid_clock.cc:648] HybridClock initialized: now 1786342827508913 us; error 33 us; skew 500 ppm
Aug 10 06:20:27 dist-test-slave-q9h9 krb5kdc[5854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342827, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:27.511339 6483 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:20:27.512079 6483 webserver.cc:460] Webserver started at http://127.31.204.195:44021/ using document root <none> and password file <none>
I20260810 06:20:27.512537 6483 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:27.512601 6483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:27.512765 6483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:27.514591 6483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "d1d95311286c415194db1dcf370f4c2a"
format_stamp: "Formatted at 2026-08-10 06:20:27 on dist-test-slave-q9h9"
server_key: "fd407d4815c7f57a4b25efcdb2a1fc2c"
server_key_iv: "ea111bbc7183f1832be630002d1d5650"
server_key_version: "encryptionkey@0"
I20260810 06:20:27.515084 6483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "d1d95311286c415194db1dcf370f4c2a"
format_stamp: "Formatted at 2026-08-10 06:20:27 on dist-test-slave-q9h9"
server_key: "fd407d4815c7f57a4b25efcdb2a1fc2c"
server_key_iv: "ea111bbc7183f1832be630002d1d5650"
server_key_version: "encryptionkey@0"
I20260810 06:20:27.518353 6483 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:20:27.520614 6502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:27.521646 6483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:27.521744 6483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "d1d95311286c415194db1dcf370f4c2a"
format_stamp: "Formatted at 2026-08-10 06:20:27 on dist-test-slave-q9h9"
server_key: "fd407d4815c7f57a4b25efcdb2a1fc2c"
server_key_iv: "ea111bbc7183f1832be630002d1d5650"
server_key_version: "encryptionkey@0"
I20260810 06:20:27.521826 6483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:27.537021 6483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:27.539325 6483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:27.539470 6483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:27.539943 6483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:27.540632 6483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:27.540675 6483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:27.540722 6483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:27.540750 6483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:27.549124 6483 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:37771
I20260810 06:20:27.549150 6691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:37771 every 8 connection(s)
I20260810 06:20:27.549855 6483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
Aug 10 06:20:27 dist-test-slave-q9h9 krb5kdc[5854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342827, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:27.559530 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 6483
I20260810 06:20:27.559620 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:20:27.559852 32563 external_mini_cluster.cc:1428] Setting key d76a57623feddf50610fc5e7988bd606
I20260810 06:20:27.561902 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:27.551371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:32961 (local address 127.31.204.254:34307)
0810 06:20:27.551489 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:27.551493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:27.552202 (+ 709us) server_negotiation.cc:407] Connection header received
0810 06:20:27.553423 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:27.553427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:27.553483 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:27.553556 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:27.554642 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:27.555164 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:27.556236 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:27.556450 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:27.556536 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:27.558673 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:27.558676 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:27.558679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:27.558705 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:27.559991 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:27.560509 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:27.560513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:27.560514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:27.560591 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:27.560902 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:27.560904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:27.560905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:27.561068 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:27.561140 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:27.561652 (+ 512us) server_negotiation.cc:299] Negotiation successful
0810 06:20:27.561768 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:20:27.562640 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:27.551650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34307 (local address 127.31.204.195:32961)
0810 06:20:27.552061 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:27.552098 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:27.553231 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:27.553751 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:27.553757 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:27.554100 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:27.554505 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:27.554517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:27.555286 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:27.555289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:27.556111 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:27.556121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:27.556580 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:27.556584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:27.556738 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:27.557207 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:27.557222 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:27.558518 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:27.560098 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:27.560104 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:27.560117 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:27.560422 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:27.560676 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:27.560679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:27.560680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:27.560811 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:27.561152 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:27.561158 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:27.561428 (+ 270us) client_negotiation.cc:769] Sending connection context
0810 06:20:27.561619 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 06:20:27.561904 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":108,"threads_started":1}
I20260810 06:20:27.563567 6692 heartbeater.cc:347] Connected to a master server at 127.31.204.254:34307
I20260810 06:20:27.563751 6692 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:27.564118 6692 heartbeater.cc:511] Master 127.31.204.254:34307 requested a full tablet report, sending...
I20260810 06:20:27.564965 5933 ts_manager.cc:194] Registered new tserver with Master: d1d95311286c415194db1dcf370f4c2a (127.31.204.195:37771)
I20260810 06:20:27.565637 5933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:32961
I20260810 06:20:27.573191 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 06:20:27 dist-test-slave-q9h9 krb5kdc[5854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 06:20:27 dist-test-slave-q9h9 krb5kdc[5854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.193@KRBTEST.COM
I20260810 06:20:27.597136 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:27.589133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54462 (local address 127.31.204.193:40673)
0810 06:20:27.589439 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:27.589448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:27.589468 (+ 20us) server_negotiation.cc:407] Connection header received
0810 06:20:27.589511 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:27.589515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:27.589666 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:27.589777 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:27.590431 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:27.591151 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:27.591888 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:27.592223 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:27.592296 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:27.594335 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:27.594344 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:27.594354 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:27.594376 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:27.595783 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:27.596095 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:27.596099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:27.596100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:27.596169 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:27.596358 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:27.596360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:27.596361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:27.596563 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:27.596694 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:27.596833 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 06:20:27.596959 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":115,"threads_started":1}
Aug 10 06:20:27 dist-test-slave-q9h9 krb5kdc[5854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:27.606865 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:27.599565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35184 (local address 127.31.204.254:34307)
0810 06:20:27.599685 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:27.599689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:27.599739 (+ 50us) server_negotiation.cc:407] Connection header received
0810 06:20:27.599820 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:27.599822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:27.599857 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:27.599917 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:27.600386 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:27.600846 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:27.601491 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:27.601684 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:27.601753 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:27.603739 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:27.603742 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:27.603744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:27.603766 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:27.605057 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:27.605579 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:27.605582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:27.605583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:27.605662 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:27.605903 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:27.605904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:27.605905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:27.606221 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:27.606342 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:27.606607 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 06:20:27.606708 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 06:20:27.607303 5933 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35184
I20260810 06:20:27.607915 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 6048
I20260810 06:20:27.636976 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 6274
I20260810 06:20:27.650403 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 6483
I20260810 06:20:27.658830 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 5875
2026-08-10T06:20:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2201 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:20:27 dist-test-slave-q9h9 krb5kdc[6705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:20:27 dist-test-slave-q9h9 krb5kdc[6705](info): set up 2 sockets
Aug 10 06:20:27 dist-test-slave-q9h9 krb5kdc[6705](info): commencing operation
krb5kdc: starting...
W20260810 06:20:28.802959 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:20:28 dist-test-slave-q9h9 krb5kdc[6705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:20:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:20:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:20:28.910905 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:38195
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:38195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:29.015811 6723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:29.016037 6723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:29.016095 6723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:29.020005 6723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:29.020056 6723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:29.020082 6723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:29.020104 6723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:29.020133 6723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:20:29.025060 6723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:38195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:38195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:29.025964 6723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:29.026899 6723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:29.034664 6733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:29.034603 6740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:29.034749 6723 server_base.cc:1034] running on GCE node
W20260810 06:20:29.034615 6738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:29.035434 6723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:29.036393 6723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:29.037544 6723 hybrid_clock.cc:648] HybridClock initialized: now 1786342829037525 us; error 32 us; skew 500 ppm
Aug 10 06:20:29 dist-test-slave-q9h9 krb5kdc[6705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342829, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:29.040097 6723 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:20:29.040733 6723 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:29.040795 6723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:29.040957 6723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:29.042882 6723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "e522b27295de43678e285becf35eb6fd"
format_stamp: "Formatted at 2026-08-10 06:20:29 on dist-test-slave-q9h9"
server_key: "8792e770bc860406637ee4e3349c6554"
server_key_iv: "160d5668c683d2c0747411cf886ebb8b"
server_key_version: "encryptionkey@0"
I20260810 06:20:29.043380 6723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "e522b27295de43678e285becf35eb6fd"
format_stamp: "Formatted at 2026-08-10 06:20:29 on dist-test-slave-q9h9"
server_key: "8792e770bc860406637ee4e3349c6554"
server_key_iv: "160d5668c683d2c0747411cf886ebb8b"
server_key_version: "encryptionkey@0"
I20260810 06:20:29.046810 6723 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260810 06:20:29.048956 6747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:29.050143 6723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:29.050261 6723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "e522b27295de43678e285becf35eb6fd"
format_stamp: "Formatted at 2026-08-10 06:20:29 on dist-test-slave-q9h9"
server_key: "8792e770bc860406637ee4e3349c6554"
server_key_iv: "160d5668c683d2c0747411cf886ebb8b"
server_key_version: "encryptionkey@0"
I20260810 06:20:29.050355 6723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:29.064267 6723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:29.067127 6723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:29.067284 6723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:29.073568 6723 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:38195
I20260810 06:20:29.073791 6840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:38195 every 8 connection(s)
I20260810 06:20:29.074184 6723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:20:29.075461 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 6723
I20260810 06:20:29.075582 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:20:29.075825 32563 external_mini_cluster.cc:1428] Setting key adb8cd5a96ac2e2c4954cec91eb64f7e
I20260810 06:20:29.076982 6841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:29.083492 6841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd: Bootstrap starting.
Aug 10 06:20:29 dist-test-slave-q9h9 krb5kdc[6705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:29.085814 6841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:29.086704 6841 log.cc:824] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:29.088825 6841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd: No bootstrap required, opened a new log
I20260810 06:20:29.089852 6844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:29.077177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.31.204.254:38195)
0810 06:20:29.077595 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:29.077613 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:29.077652 (+ 39us) server_negotiation.cc:407] Connection header received
0810 06:20:29.078483 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:29.078504 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:29.078883 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:29.079212 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:29.079848 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.081245 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.082053 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.082733 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.082915 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:29.085192 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:29.085198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:29.085212 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:29.085242 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:29.087088 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:29.087544 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:29.087549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:29.087555 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:29.087684 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:29.087911 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:29.087914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:29.087915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:29.088367 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:29.088544 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:29.088718 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 06:20:29.088983 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":110,"threads_started":1}
I20260810 06:20:29.091873 6841 raft_consensus.cc:348] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e522b27295de43678e285becf35eb6fd" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38195 } }
I20260810 06:20:29.092067 6841 raft_consensus.cc:374] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:29.092164 6841 raft_consensus.cc:729] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e522b27295de43678e285becf35eb6fd, State: Initialized, Role: FOLLOWER
I20260810 06:20:29.092553 6841 consensus_queue.cc:260] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e522b27295de43678e285becf35eb6fd" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38195 } }
I20260810 06:20:29.092669 6841 raft_consensus.cc:388] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:20:29.092711 6841 raft_consensus.cc:482] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:20:29.092782 6841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:29.093564 6841 raft_consensus.cc:504] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e522b27295de43678e285becf35eb6fd" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38195 } }
I20260810 06:20:29.093842 6841 leader_election.cc:302] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e522b27295de43678e285becf35eb6fd; no voters:
I20260810 06:20:29.094100 6841 leader_election.cc:288] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:20:29.094208 6846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:29.094437 6846 raft_consensus.cc:686] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd [term 1 LEADER]: Becoming Leader. State: Replica: e522b27295de43678e285becf35eb6fd, State: Running, Role: LEADER
I20260810 06:20:29.094789 6846 consensus_queue.cc:237] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e522b27295de43678e285becf35eb6fd" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38195 } }
I20260810 06:20:29.095294 6841 sys_catalog.cc:564] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:29.096367 6849 sys_catalog.cc:455] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e522b27295de43678e285becf35eb6fd. Latest consensus state: current_term: 1 leader_uuid: "e522b27295de43678e285becf35eb6fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e522b27295de43678e285becf35eb6fd" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38195 } } }
I20260810 06:20:29.096484 6849 sys_catalog.cc:458] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:29.096750 6847 sys_catalog.cc:455] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e522b27295de43678e285becf35eb6fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e522b27295de43678e285becf35eb6fd" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38195 } } }
I20260810 06:20:29.096830 6847 sys_catalog.cc:458] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:29.096880 6858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:29.099488 6858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:29.104965 6858 catalog_manager.cc:1347] Generated new cluster ID: a845d4c0633f454fb2b5857633b72234
I20260810 06:20:29.105026 6858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:29.124383 6858 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:20:29.125587 6858 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:29.138859 6858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd: Generated new TSK 0
I20260810 06:20:29.139562 6858 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
I20260810 06:20:29.168552 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:38195
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:29.275791 6875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:29.276014 6875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:29.276054 6875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:29.279884 6875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:29.279928 6875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:29.279983 6875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:20:29.284936 6875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:38195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:29.285784 6875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:29.286695 6875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:20:29.295792 6875 server_base.cc:1034] running on GCE node
W20260810 06:20:29.297271 6890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:29.297510 6883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:29.297698 6887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:29.298137 6875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:29.298730 6875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:29.299892 6875 hybrid_clock.cc:648] HybridClock initialized: now 1786342829299872 us; error 31 us; skew 500 ppm
Aug 10 06:20:29 dist-test-slave-q9h9 krb5kdc[6705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342829, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:29.302569 6875 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:20:29.303225 6875 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:29.303295 6875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:29.303483 6875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:29.305393 6875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "ae55961b03374468b40b406955715da8"
format_stamp: "Formatted at 2026-08-10 06:20:29 on dist-test-slave-q9h9"
server_key: "58ddc58deaf2b252f4b1cd3c0130fc1e"
server_key_iv: "88f9b988619b567f8a7b7f38d539d8e2"
server_key_version: "encryptionkey@0"
I20260810 06:20:29.305915 6875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "ae55961b03374468b40b406955715da8"
format_stamp: "Formatted at 2026-08-10 06:20:29 on dist-test-slave-q9h9"
server_key: "58ddc58deaf2b252f4b1cd3c0130fc1e"
server_key_iv: "88f9b988619b567f8a7b7f38d539d8e2"
server_key_version: "encryptionkey@0"
I20260810 06:20:29.309267 6875 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 06:20:29.311462 6898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:29.312557 6875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:29.312664 6875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "ae55961b03374468b40b406955715da8"
format_stamp: "Formatted at 2026-08-10 06:20:29 on dist-test-slave-q9h9"
server_key: "58ddc58deaf2b252f4b1cd3c0130fc1e"
server_key_iv: "88f9b988619b567f8a7b7f38d539d8e2"
server_key_version: "encryptionkey@0"
I20260810 06:20:29.312762 6875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:29.327368 6875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:29.330271 6875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:29.330430 6875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:29.330935 6875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:29.331688 6875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:29.331737 6875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:29.331786 6875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:29.331816 6875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:29.340449 6875 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:40665
I20260810 06:20:29.340466 7098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:40665 every 8 connection(s)
I20260810 06:20:29.340960 6875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:20:29.343935 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 6875
I20260810 06:20:29.344065 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:20:29.344372 32563 external_mini_cluster.cc:1428] Setting key 72f7efa7c0d89878de9be7162b1ad634
Aug 10 06:20:29 dist-test-slave-q9h9 krb5kdc[6705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342829, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.204.194@KRBTEST.COM; defaulting to no policy
I20260810 06:20:29.354390 6844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:29.342556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:48129 (local address 127.31.204.254:38195)
0810 06:20:29.342695 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:29.342700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:29.343500 (+ 800us) server_negotiation.cc:407] Connection header received
0810 06:20:29.344969 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:29.344972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:29.345020 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:29.345095 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:29.346336 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.346863 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.347944 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.348180 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.348274 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:29.350868 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:29.350871 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:29.350873 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:29.350898 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:29.352193 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:29.352775 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:29.352778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:29.352779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:29.352859 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:29.353231 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:29.353234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:29.353235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:29.353401 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:29.353474 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:29.354167 (+ 693us) server_negotiation.cc:299] Negotiation successful
0810 06:20:29.354274 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 06:20:29.355154 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:29.342921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:38195 (local address 127.31.204.193:48129)
0810 06:20:29.343357 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:29.343396 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:29.344754 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:29.345278 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:29.345284 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:29.345665 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:29.346192 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:29.346209 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.346983 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.346986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:29.347824 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:29.347833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.348321 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.348325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:29.348481 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:29.348972 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:29.348988 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:29.350680 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:29.352318 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:29.352324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:29.352338 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:29.352664 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:29.352945 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:29.352948 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:29.352950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:29.353122 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:29.353510 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:29.353516 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:29.353848 (+ 332us) client_negotiation.cc:769] Sending connection context
0810 06:20:29.354082 (+ 234us) client_negotiation.cc:240] Negotiation successful
0810 06:20:29.354389 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":115,"threads_started":1}
I20260810 06:20:29.356340 7099 heartbeater.cc:347] Connected to a master server at 127.31.204.254:38195
I20260810 06:20:29.356535 7099 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.194@KRBTEST.COM" created.
I20260810 06:20:29.357028 7099 heartbeater.cc:511] Master 127.31.204.254:38195 requested a full tablet report, sending...
I20260810 06:20:29.358418 6776 ts_manager.cc:194] Registered new tserver with Master: ae55961b03374468b40b406955715da8 (127.31.204.193:40665)
I20260810 06:20:29.359558 6776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:48129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
I20260810 06:20:29.372395 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:38195
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:29.478596 7107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:29.478806 7107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:29.478852 7107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:29.482667 7107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:29.482712 7107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:29.482782 7107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:20:29.487669 7107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:38195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:29.488514 7107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:29.489410 7107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:29.497802 7117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:29.497807 7114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:29.497815 7120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:29.498078 7107 server_base.cc:1034] running on GCE node
I20260810 06:20:29.498548 7107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:29.499125 7107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:29.500273 7107 hybrid_clock.cc:648] HybridClock initialized: now 1786342829500259 us; error 27 us; skew 500 ppm
Aug 10 06:20:29 dist-test-slave-q9h9 krb5kdc[6705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342829, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:29.502835 7107 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:20:29.503489 7107 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:29.503552 7107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:29.503732 7107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:29.505517 7107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "8fb6f0d7a0a14549a748e33a65e38b77"
format_stamp: "Formatted at 2026-08-10 06:20:29 on dist-test-slave-q9h9"
server_key: "e5e882b79360aec23bbf023611b7ad06"
server_key_iv: "4a42f596103c357ce7adab11924ccc60"
server_key_version: "encryptionkey@0"
I20260810 06:20:29.506036 7107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "8fb6f0d7a0a14549a748e33a65e38b77"
format_stamp: "Formatted at 2026-08-10 06:20:29 on dist-test-slave-q9h9"
server_key: "e5e882b79360aec23bbf023611b7ad06"
server_key_iv: "4a42f596103c357ce7adab11924ccc60"
server_key_version: "encryptionkey@0"
I20260810 06:20:29.509286 7107 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:20:29.511437 7131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:29.512485 7107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:29.512580 7107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "8fb6f0d7a0a14549a748e33a65e38b77"
format_stamp: "Formatted at 2026-08-10 06:20:29 on dist-test-slave-q9h9"
server_key: "e5e882b79360aec23bbf023611b7ad06"
server_key_iv: "4a42f596103c357ce7adab11924ccc60"
server_key_version: "encryptionkey@0"
I20260810 06:20:29.512673 7107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:29.537017 7107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:29.539805 7107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:29.539958 7107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:29.540458 7107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:29.541220 7107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:29.541266 7107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:29.541302 7107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:29.541322 7107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:29.551515 7107 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:35059
I20260810 06:20:29.551535 7310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:35059 every 8 connection(s)
I20260810 06:20:29.552008 7107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:20:29.557919 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 7107
I20260810 06:20:29.558059 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:20:29.558336 32563 external_mini_cluster.cc:1428] Setting key cfc2a89db94a84e81195281c3b9d872c
Aug 10 06:20:29 dist-test-slave-q9h9 krb5kdc[6705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342829, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:29.563970 6844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:29.553486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:50741 (local address 127.31.204.254:38195)
0810 06:20:29.553599 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:29.553603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:29.554377 (+ 774us) server_negotiation.cc:407] Connection header received
0810 06:20:29.555630 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:29.555632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:29.555679 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:29.555741 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:29.556713 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.557225 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.558340 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.558605 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.558694 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:29.560883 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:29.560886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:29.560888 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:29.560910 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:29.562191 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:29.562637 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:29.562641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:29.562643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:29.562718 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:29.563014 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:29.563018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:29.563019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:29.563182 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:29.563275 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:29.563702 (+ 427us) server_negotiation.cc:299] Negotiation successful
0810 06:20:29.563820 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 06:20:29.564625 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:29.553808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:38195 (local address 127.31.204.194:50741)
0810 06:20:29.554244 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:29.554285 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:29.555441 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:29.555878 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:29.555884 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:29.556190 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:29.556608 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:29.556620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.557311 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.557313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:29.558231 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:29.558241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.558750 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.558754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:29.558915 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:29.559399 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:29.559418 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:29.560727 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:29.562296 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:29.562301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:29.562314 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:29.562546 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:29.562803 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:29.562806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:29.562808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:29.562930 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:29.563278 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:29.563284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:29.563493 (+ 209us) client_negotiation.cc:769] Sending connection context
0810 06:20:29.563642 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 06:20:29.563919 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":107,"threads_started":1}
I20260810 06:20:29.565601 7311 heartbeater.cc:347] Connected to a master server at 127.31.204.254:38195
I20260810 06:20:29.565793 7311 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:29.566195 7311 heartbeater.cc:511] Master 127.31.204.254:38195 requested a full tablet report, sending...
I20260810 06:20:29.567131 6776 ts_manager.cc:194] Registered new tserver with Master: 8fb6f0d7a0a14549a748e33a65e38b77 (127.31.204.194:35059)
WARNING: no policy specified for kudu/127.31.204.195@KRBTEST.COM; defaulting to no policy
I20260810 06:20:29.567869 6776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:50741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
I20260810 06:20:29.582319 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:38195
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:29.690033 7317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:29.690239 7317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:29.690279 7317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:29.694088 7317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:29.694137 7317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:29.694216 7317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:20:29.699244 7317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:38195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:29.700066 7317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:29.700917 7317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:29.709060 7325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:29.709065 7328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:29.709220 7323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:29.709338 7317 server_base.cc:1034] running on GCE node
I20260810 06:20:29.709726 7317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:29.710296 7317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:29.711441 7317 hybrid_clock.cc:648] HybridClock initialized: now 1786342829711416 us; error 48 us; skew 500 ppm
Aug 10 06:20:29 dist-test-slave-q9h9 krb5kdc[6705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342829, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:29.717612 7317 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:20:29.718297 7317 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:29.718366 7317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:29.718554 7317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:29.720383 7317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "f37d89ab5ff74ef49387eca385b85c66"
format_stamp: "Formatted at 2026-08-10 06:20:29 on dist-test-slave-q9h9"
server_key: "edb931eb06ef4d2c03742a92cdcf54c1"
server_key_iv: "66dd1c42038a4c799029cdaeb61d43fb"
server_key_version: "encryptionkey@0"
I20260810 06:20:29.720970 7317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "f37d89ab5ff74ef49387eca385b85c66"
format_stamp: "Formatted at 2026-08-10 06:20:29 on dist-test-slave-q9h9"
server_key: "edb931eb06ef4d2c03742a92cdcf54c1"
server_key_iv: "66dd1c42038a4c799029cdaeb61d43fb"
server_key_version: "encryptionkey@0"
I20260810 06:20:29.724200 7317 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 06:20:29.726294 7335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:29.727347 7317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:20:29.727448 7317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "f37d89ab5ff74ef49387eca385b85c66"
format_stamp: "Formatted at 2026-08-10 06:20:29 on dist-test-slave-q9h9"
server_key: "edb931eb06ef4d2c03742a92cdcf54c1"
server_key_iv: "66dd1c42038a4c799029cdaeb61d43fb"
server_key_version: "encryptionkey@0"
I20260810 06:20:29.727531 7317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:29.742478 7317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:29.744786 7317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:29.744933 7317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:29.745409 7317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:29.746147 7317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:29.746191 7317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:29.746240 7317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:29.746269 7317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:29.754283 7317 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:39711
I20260810 06:20:29.754307 7542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:39711 every 8 connection(s)
I20260810 06:20:29.754758 7317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:20:29.757510 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 7317
I20260810 06:20:29.757632 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:20:29.757932 32563 external_mini_cluster.cc:1428] Setting key c7931bc12cc56706295e00b8e7e57eeb
Aug 10 06:20:29 dist-test-slave-q9h9 krb5kdc[6705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342829, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:29.766570 6844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:29.756233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:46357 (local address 127.31.204.254:38195)
0810 06:20:29.756347 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:29.756350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:29.757069 (+ 719us) server_negotiation.cc:407] Connection header received
0810 06:20:29.758484 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:29.758486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:29.758535 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:29.758615 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:29.759712 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.760265 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.761267 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.761471 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.761536 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:29.763641 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:29.763644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:29.763646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:29.763668 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:29.764749 (+ 1081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:29.765190 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:29.765193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:29.765195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:29.765268 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:29.765561 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:29.765564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:29.765566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:29.765737 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:29.765819 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:29.766304 (+ 485us) server_negotiation.cc:299] Negotiation successful
0810 06:20:29.766419 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:20:29.767206 7546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:29.756525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:38195 (local address 127.31.204.195:46357)
0810 06:20:29.756929 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:29.756967 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:29.758297 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:29.758769 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:29.758777 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:29.759118 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:29.759576 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:29.759589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.760366 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.760368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:29.761178 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:29.761187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.761566 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.761569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:29.761692 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:29.762159 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:29.762175 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:29.763494 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:29.764853 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:29.764859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:29.764871 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:29.765098 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:29.765350 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:29.765353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:29.765355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:29.765475 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:29.765834 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:29.765841 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:29.766077 (+ 236us) client_negotiation.cc:769] Sending connection context
0810 06:20:29.766247 (+ 170us) client_negotiation.cc:240] Negotiation successful
0810 06:20:29.766518 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":109,"threads_started":1}
I20260810 06:20:29.768096 7543 heartbeater.cc:347] Connected to a master server at 127.31.204.254:38195
I20260810 06:20:29.768294 7543 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:29.768653 7543 heartbeater.cc:511] Master 127.31.204.254:38195 requested a full tablet report, sending...
I20260810 06:20:29.769498 6776 ts_manager.cc:194] Registered new tserver with Master: f37d89ab5ff74ef49387eca385b85c66 (127.31.204.195:39711)
I20260810 06:20:29.769989 6776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:46357
I20260810 06:20:29.771293 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:20:29.779758 6844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:29.773416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.31.204.254:38195)
0810 06:20:29.773606 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:29.773610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:29.773730 (+ 120us) server_negotiation.cc:407] Connection header received
0810 06:20:29.773920 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:29.773923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:29.773970 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:29.774071 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:29.774909 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.775387 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.776142 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.776345 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.776409 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:29.777451 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:29.777454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:29.777455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:29.777476 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:29.778564 (+ 1088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:29.778915 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:29.778917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:29.778918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:29.778978 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:29.779152 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:29.779154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:29.779155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:29.779342 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:29.779442 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:29.779572 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 06:20:29.779656 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 06:20:29.782238 6776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:20:29.785079 6776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:20:29.798264 7559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:29.792559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:39711 (local address 127.0.0.1:58626)
0810 06:20:29.793055 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:29.793101 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:29.793222 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:29.793971 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:29.793977 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:29.793999 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:29.794141 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:29.794148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.795511 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.795514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:29.796747 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:29.796756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.797749 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.797753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:29.797853 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:29.797888 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 06:20:29.797991 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 06:20:29.798126 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":71,"thread_start_us":62,"threads_started":1}
I20260810 06:20:29.798300 7565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:29.793183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58626 (local address 127.31.204.195:39711)
0810 06:20:29.793526 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:29.793537 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:29.793558 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:20:29.793625 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:29.793629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:29.793826 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:29.793972 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:29.794253 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.795369 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.796891 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.797629 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.797746 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:29.797864 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:29.798052 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 06:20:29.798141 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":124,"threads_started":1}
I20260810 06:20:29.798519 7563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:29.792869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40844 (local address 127.31.204.193:40665)
0810 06:20:29.793268 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:29.793280 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:29.793549 (+ 269us) server_negotiation.cc:407] Connection header received
0810 06:20:29.793792 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:29.793798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:29.793985 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:29.794133 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:29.794557 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.795428 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.797073 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.797701 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.797790 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:29.797910 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:29.798273 (+ 363us) server_negotiation.cc:299] Negotiation successful
0810 06:20:29.798368 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":102,"threads_started":1}
I20260810 06:20:29.798780 7557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:29.793207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:35059 (local address 127.0.0.1:42264)
0810 06:20:29.793681 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:29.793693 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:29.793758 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:29.794818 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:29.794821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:29.794842 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:29.794928 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:29.794933 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.796164 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.796168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:29.797235 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:29.797245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.798419 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.798423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:29.798512 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:29.798533 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 06:20:29.798597 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 06:20:29.798665 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":98,"threads_started":1}
I20260810 06:20:29.798894 7564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:29.792712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:40665 (local address 127.0.0.1:40844)
0810 06:20:29.793447 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:29.793461 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:29.793638 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:29.794328 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:29.794331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:29.794349 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:29.794439 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:29.794446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.795561 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.795564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:29.796941 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:29.796953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.798005 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.798008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:29.798120 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:29.798140 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 06:20:29.798654 (+ 514us) client_negotiation.cc:240] Negotiation successful
0810 06:20:29.798730 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":72,"thread_start_us":104,"threads_started":1}
I20260810 06:20:29.799229 7566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:29.793627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42264 (local address 127.31.204.194:35059)
0810 06:20:29.794413 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:29.794429 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:29.794449 (+ 20us) server_negotiation.cc:407] Connection header received
0810 06:20:29.794515 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:29.794519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:29.794676 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:29.794836 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:29.795030 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.796027 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.797355 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.798130 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.798241 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:29.798350 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:29.798998 (+ 648us) server_negotiation.cc:299] Negotiation successful
0810 06:20:29.799086 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":144,"threads_started":1}
I20260810 06:20:29.801314 6979 tablet_service.cc:1467] Processing CreateTablet for tablet 36305bef78eb4cf28f6f2ed951dbf251 (DEFAULT_TABLE table=test-table [id=4eec9162554f4a2598fe1c59bd19f898]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:29.801509 7425 tablet_service.cc:1467] Processing CreateTablet for tablet 36305bef78eb4cf28f6f2ed951dbf251 (DEFAULT_TABLE table=test-table [id=4eec9162554f4a2598fe1c59bd19f898]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:29.801509 7203 tablet_service.cc:1467] Processing CreateTablet for tablet 36305bef78eb4cf28f6f2ed951dbf251 (DEFAULT_TABLE table=test-table [id=4eec9162554f4a2598fe1c59bd19f898]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:29.802354 6979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36305bef78eb4cf28f6f2ed951dbf251. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:29.802518 7203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36305bef78eb4cf28f6f2ed951dbf251. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:29.802525 7425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36305bef78eb4cf28f6f2ed951dbf251. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:29.808104 7567 tablet_bootstrap.cc:492] T 36305bef78eb4cf28f6f2ed951dbf251 P ae55961b03374468b40b406955715da8: Bootstrap starting.
I20260810 06:20:29.809732 7570 tablet_bootstrap.cc:492] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77: Bootstrap starting.
I20260810 06:20:29.809732 7573 tablet_bootstrap.cc:492] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66: Bootstrap starting.
I20260810 06:20:29.810524 7567 tablet_bootstrap.cc:654] T 36305bef78eb4cf28f6f2ed951dbf251 P ae55961b03374468b40b406955715da8: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:29.811731 7567 log.cc:824] T 36305bef78eb4cf28f6f2ed951dbf251 P ae55961b03374468b40b406955715da8: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:29.811805 7570 tablet_bootstrap.cc:654] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:29.811805 7573 tablet_bootstrap.cc:654] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:29.813122 7573 log.cc:824] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:29.813112 7570 log.cc:824] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:29.813383 7567 tablet_bootstrap.cc:492] T 36305bef78eb4cf28f6f2ed951dbf251 P ae55961b03374468b40b406955715da8: No bootstrap required, opened a new log
I20260810 06:20:29.813565 7567 ts_tablet_manager.cc:1397] T 36305bef78eb4cf28f6f2ed951dbf251 P ae55961b03374468b40b406955715da8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:20:29.814970 7570 tablet_bootstrap.cc:492] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77: No bootstrap required, opened a new log
I20260810 06:20:29.815074 7573 tablet_bootstrap.cc:492] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66: No bootstrap required, opened a new log
I20260810 06:20:29.815157 7570 ts_tablet_manager.cc:1397] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:20:29.815271 7573 ts_tablet_manager.cc:1397] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 06:20:29.817320 7567 raft_consensus.cc:348] T 36305bef78eb4cf28f6f2ed951dbf251 P ae55961b03374468b40b406955715da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37d89ab5ff74ef49387eca385b85c66" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39711 } } peers { permanent_uuid: "ae55961b03374468b40b406955715da8" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40665 } } peers { permanent_uuid: "8fb6f0d7a0a14549a748e33a65e38b77" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 35059 } }
I20260810 06:20:29.817559 7567 raft_consensus.cc:374] T 36305bef78eb4cf28f6f2ed951dbf251 P ae55961b03374468b40b406955715da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:29.817605 7567 raft_consensus.cc:729] T 36305bef78eb4cf28f6f2ed951dbf251 P ae55961b03374468b40b406955715da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae55961b03374468b40b406955715da8, State: Initialized, Role: FOLLOWER
I20260810 06:20:29.818048 7567 consensus_queue.cc:260] T 36305bef78eb4cf28f6f2ed951dbf251 P ae55961b03374468b40b406955715da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37d89ab5ff74ef49387eca385b85c66" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39711 } } peers { permanent_uuid: "ae55961b03374468b40b406955715da8" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40665 } } peers { permanent_uuid: "8fb6f0d7a0a14549a748e33a65e38b77" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 35059 } }
I20260810 06:20:29.818063 7570 raft_consensus.cc:348] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37d89ab5ff74ef49387eca385b85c66" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39711 } } peers { permanent_uuid: "ae55961b03374468b40b406955715da8" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40665 } } peers { permanent_uuid: "8fb6f0d7a0a14549a748e33a65e38b77" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 35059 } }
I20260810 06:20:29.818261 7570 raft_consensus.cc:374] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:29.818292 7570 raft_consensus.cc:729] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fb6f0d7a0a14549a748e33a65e38b77, State: Initialized, Role: FOLLOWER
I20260810 06:20:29.818653 7570 consensus_queue.cc:260] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37d89ab5ff74ef49387eca385b85c66" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39711 } } peers { permanent_uuid: "ae55961b03374468b40b406955715da8" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40665 } } peers { permanent_uuid: "8fb6f0d7a0a14549a748e33a65e38b77" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 35059 } }
I20260810 06:20:29.818701 7099 heartbeater.cc:503] Master 127.31.204.254:38195 was elected leader, sending a full tablet report...
I20260810 06:20:29.818822 7573 raft_consensus.cc:348] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37d89ab5ff74ef49387eca385b85c66" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39711 } } peers { permanent_uuid: "ae55961b03374468b40b406955715da8" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40665 } } peers { permanent_uuid: "8fb6f0d7a0a14549a748e33a65e38b77" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 35059 } }
I20260810 06:20:29.819043 7573 raft_consensus.cc:374] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:29.819088 7573 raft_consensus.cc:729] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37d89ab5ff74ef49387eca385b85c66, State: Initialized, Role: FOLLOWER
I20260810 06:20:29.819175 7311 heartbeater.cc:503] Master 127.31.204.254:38195 was elected leader, sending a full tablet report...
I20260810 06:20:29.819469 7573 consensus_queue.cc:260] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37d89ab5ff74ef49387eca385b85c66" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39711 } } peers { permanent_uuid: "ae55961b03374468b40b406955715da8" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40665 } } peers { permanent_uuid: "8fb6f0d7a0a14549a748e33a65e38b77" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 35059 } }
I20260810 06:20:29.820007 7567 ts_tablet_manager.cc:1428] T 36305bef78eb4cf28f6f2ed951dbf251 P ae55961b03374468b40b406955715da8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 06:20:29.820209 7543 heartbeater.cc:503] Master 127.31.204.254:38195 was elected leader, sending a full tablet report...
I20260810 06:20:29.820348 7573 ts_tablet_manager.cc:1428] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 06:20:29.821297 7570 ts_tablet_manager.cc:1428] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
W20260810 06:20:29.841943 7101 tablet.cc:2369] T 36305bef78eb4cf28f6f2ed951dbf251 P ae55961b03374468b40b406955715da8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:20:29.915788 7579 raft_consensus.cc:482] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:20:29.915983 7579 raft_consensus.cc:504] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37d89ab5ff74ef49387eca385b85c66" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39711 } } peers { permanent_uuid: "ae55961b03374468b40b406955715da8" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40665 } } peers { permanent_uuid: "8fb6f0d7a0a14549a748e33a65e38b77" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 35059 } }
I20260810 06:20:29.917426 7579 leader_election.cc:288] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f37d89ab5ff74ef49387eca385b85c66 (127.31.204.195:39711), ae55961b03374468b40b406955715da8 (127.31.204.193:40665)
I20260810 06:20:29.920573 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:29.917392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:39711 (local address 127.31.204.194:53101)
0810 06:20:29.917573 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:29.917588 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:29.917674 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:29.917931 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:29.917934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:29.917954 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:29.918101 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:29.918107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.918802 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.918805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:29.919567 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:29.919575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.920185 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.920189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:29.920291 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:29.920313 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:20:29.920380 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 06:20:29.920448 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260810 06:20:29.920573 7565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:29.917512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:53101 (local address 127.31.204.195:39711)
0810 06:20:29.917634 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:29.917638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:29.917653 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:20:29.917793 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:29.917796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:29.917834 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:29.917904 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:29.918204 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.918696 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.919691 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.920082 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.920185 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:29.920237 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:29.920407 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 06:20:29.920473 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:20:29.920965 7581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:29.917397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:40665 (local address 127.31.204.194:47035)
0810 06:20:29.917945 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:29.917956 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:29.918067 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:29.918313 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:29.918316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:29.918332 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:29.918446 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:29.918451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.919134 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.919137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:29.919900 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:29.919908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.920518 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.920521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:29.920618 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:29.920641 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 06:20:29.920764 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 06:20:29.920828 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":205,"thread_start_us":135,"threads_started":1}
I20260810 06:20:29.920989 7563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:29.917462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:47035 (local address 127.31.204.193:40665)
0810 06:20:29.917576 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:29.917580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:29.918042 (+ 462us) server_negotiation.cc:407] Connection header received
0810 06:20:29.918170 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:29.918173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:29.918225 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:29.918287 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:29.918552 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.919027 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.920023 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.920408 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.920509 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:29.920558 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:29.920799 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 06:20:29.920878 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:20:29.921247 7461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36305bef78eb4cf28f6f2ed951dbf251" candidate_uuid: "8fb6f0d7a0a14549a748e33a65e38b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37d89ab5ff74ef49387eca385b85c66" is_pre_election: true
I20260810 06:20:29.921546 7461 raft_consensus.cc:2454] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fb6f0d7a0a14549a748e33a65e38b77 in term 0.
I20260810 06:20:29.921662 7016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36305bef78eb4cf28f6f2ed951dbf251" candidate_uuid: "8fb6f0d7a0a14549a748e33a65e38b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae55961b03374468b40b406955715da8" is_pre_election: true
I20260810 06:20:29.921923 7016 raft_consensus.cc:2454] T 36305bef78eb4cf28f6f2ed951dbf251 P ae55961b03374468b40b406955715da8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fb6f0d7a0a14549a748e33a65e38b77 in term 0.
I20260810 06:20:29.922004 7136 leader_election.cc:302] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fb6f0d7a0a14549a748e33a65e38b77, f37d89ab5ff74ef49387eca385b85c66; no voters:
I20260810 06:20:29.922298 7579 raft_consensus.cc:2781] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:20:29.922376 7579 raft_consensus.cc:482] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:20:29.922407 7579 raft_consensus.cc:3037] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:29.923409 7579 raft_consensus.cc:504] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37d89ab5ff74ef49387eca385b85c66" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39711 } } peers { permanent_uuid: "ae55961b03374468b40b406955715da8" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40665 } } peers { permanent_uuid: "8fb6f0d7a0a14549a748e33a65e38b77" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 35059 } }
I20260810 06:20:29.923831 7579 leader_election.cc:288] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [CANDIDATE]: Term 1 election: Requested vote from peers f37d89ab5ff74ef49387eca385b85c66 (127.31.204.195:39711), ae55961b03374468b40b406955715da8 (127.31.204.193:40665)
I20260810 06:20:29.924124 7461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36305bef78eb4cf28f6f2ed951dbf251" candidate_uuid: "8fb6f0d7a0a14549a748e33a65e38b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37d89ab5ff74ef49387eca385b85c66"
I20260810 06:20:29.924191 7016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36305bef78eb4cf28f6f2ed951dbf251" candidate_uuid: "8fb6f0d7a0a14549a748e33a65e38b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae55961b03374468b40b406955715da8"
I20260810 06:20:29.924242 7461 raft_consensus.cc:3037] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:29.924307 7016 raft_consensus.cc:3037] T 36305bef78eb4cf28f6f2ed951dbf251 P ae55961b03374468b40b406955715da8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:29.925246 7461 raft_consensus.cc:2454] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fb6f0d7a0a14549a748e33a65e38b77 in term 1.
I20260810 06:20:29.925277 7016 raft_consensus.cc:2454] T 36305bef78eb4cf28f6f2ed951dbf251 P ae55961b03374468b40b406955715da8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fb6f0d7a0a14549a748e33a65e38b77 in term 1.
I20260810 06:20:29.925521 7136 leader_election.cc:302] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fb6f0d7a0a14549a748e33a65e38b77, f37d89ab5ff74ef49387eca385b85c66; no voters:
I20260810 06:20:29.925710 7579 raft_consensus.cc:2781] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:29.925907 7579 raft_consensus.cc:686] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [term 1 LEADER]: Becoming Leader. State: Replica: 8fb6f0d7a0a14549a748e33a65e38b77, State: Running, Role: LEADER
I20260810 06:20:29.926213 7579 consensus_queue.cc:237] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37d89ab5ff74ef49387eca385b85c66" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39711 } } peers { permanent_uuid: "ae55961b03374468b40b406955715da8" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40665 } } peers { permanent_uuid: "8fb6f0d7a0a14549a748e33a65e38b77" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 35059 } }
I20260810 06:20:29.929144 6769 catalog_manager.cc:5495] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8fb6f0d7a0a14549a748e33a65e38b77 (127.31.204.194). New cstate: current_term: 1 leader_uuid: "8fb6f0d7a0a14549a748e33a65e38b77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f37d89ab5ff74ef49387eca385b85c66" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae55961b03374468b40b406955715da8" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fb6f0d7a0a14549a748e33a65e38b77" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 35059 } health_report { overall_health: HEALTHY } } }
I20260810 06:20:29.948355 7566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:29.945444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42274 (local address 127.31.204.194:35059)
0810 06:20:29.945554 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:29.945557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:29.945608 (+ 51us) server_negotiation.cc:407] Connection header received
0810 06:20:29.945734 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:29.945736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:29.945779 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:29.945856 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:29.946072 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.946518 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.947246 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:29.947435 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:29.947497 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:29.947712 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:29.948057 (+ 345us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:29.948139 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:29.948183 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 06:20:29.948223 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:20:29.953338 7461 raft_consensus.cc:1261] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66 [term 1 FOLLOWER]: Refusing update from remote peer 8fb6f0d7a0a14549a748e33a65e38b77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:29.953369 7016 raft_consensus.cc:1261] T 36305bef78eb4cf28f6f2ed951dbf251 P ae55961b03374468b40b406955715da8 [term 1 FOLLOWER]: Refusing update from remote peer 8fb6f0d7a0a14549a748e33a65e38b77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:29.953944 7582 consensus_queue.cc:1035] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae55961b03374468b40b406955715da8" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:29.954139 7579 consensus_queue.cc:1035] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f37d89ab5ff74ef49387eca385b85c66" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:29.960861 7587 mvcc.cc:205] Tried to move back new op lower bound from 7316860231482994688 to 7316860231380062208. Current Snapshot: MvccSnapshot[applied={T|T < 7316860231482994688}]
I20260810 06:20:29.965306 7590 mvcc.cc:205] Tried to move back new op lower bound from 7316860231482994688 to 7316860231380062208. Current Snapshot: MvccSnapshot[applied={T|T < 7316860231482994688 or (T in {7316860231482994688})}]
I20260810 06:20:29.968752 6772 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:29.968879 6772 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:29.971024 6772 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e522b27295de43678e285becf35eb6fd: Sending DeleteTablet for 3 replicas of tablet 36305bef78eb4cf28f6f2ed951dbf251
I20260810 06:20:29.971695 7425 tablet_service.cc:1514] Processing DeleteTablet for tablet 36305bef78eb4cf28f6f2ed951dbf251 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:29 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:58626
I20260810 06:20:29.971939 7203 tablet_service.cc:1514] Processing DeleteTablet for tablet 36305bef78eb4cf28f6f2ed951dbf251 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:29 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:42264
I20260810 06:20:29.972143 7605 tablet_replica.cc:330] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66: stopping tablet replica
I20260810 06:20:29.972203 6979 tablet_service.cc:1514] Processing DeleteTablet for tablet 36305bef78eb4cf28f6f2ed951dbf251 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:29 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:40844
I20260810 06:20:29.972478 7605 raft_consensus.cc:2229] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:29.972752 7605 raft_consensus.cc:2258] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:29.972829 7606 tablet_replica.cc:330] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77: stopping tablet replica
I20260810 06:20:29.972607 7607 tablet_replica.cc:330] T 36305bef78eb4cf28f6f2ed951dbf251 P ae55961b03374468b40b406955715da8: stopping tablet replica
I20260810 06:20:29.973420 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 6875
I20260810 06:20:29.973438 7606 raft_consensus.cc:2229] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:20:29.999457 7606 raft_consensus.cc:2258] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:30.002241 7606 ts_tablet_manager.cc:1905] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:30.005973 7606 ts_tablet_manager.cc:1918] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:20:30.006121 7606 log.cc:1196] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/36305bef78eb4cf28f6f2ed951dbf251
I20260810 06:20:30.006536 7606 ts_tablet_manager.cc:1939] T 36305bef78eb4cf28f6f2ed951dbf251 P 8fb6f0d7a0a14549a748e33a65e38b77: Deleting consensus metadata
W20260810 06:20:30.013871 6751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.204.193:40665 (error 108)
I20260810 06:20:30.014391 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 7107
I20260810 06:20:30.014595 6751 catalog_manager.cc:4841] TS 8fb6f0d7a0a14549a748e33a65e38b77 (127.31.204.194:35059): tablet 36305bef78eb4cf28f6f2ed951dbf251 (table test-table [id=4eec9162554f4a2598fe1c59bd19f898]) successfully deleted
I20260810 06:20:30.027912 7605 ts_tablet_manager.cc:1905] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:30.030712 7605 ts_tablet_manager.cc:1918] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:20:30.030803 7605 log.cc:1196] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/36305bef78eb4cf28f6f2ed951dbf251
I20260810 06:20:30.031118 7605 ts_tablet_manager.cc:1939] T 36305bef78eb4cf28f6f2ed951dbf251 P f37d89ab5ff74ef49387eca385b85c66: Deleting consensus metadata
I20260810 06:20:30.031822 7557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.031493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:40665 (local address 127.0.0.1:40846)
0810 06:20:30.031643 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:30.031713 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:40665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 06:20:30.032027 6751 catalog_manager.cc:4584] TS ae55961b03374468b40b406955715da8 (127.31.204.193:40665): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 36305bef78eb4cf28f6f2ed951dbf251: Network error: Client connection negotiation failed: client connection to 127.31.204.193:40665: connect: Connection refused (error 111)
I20260810 06:20:30.032414 6752 catalog_manager.cc:4841] TS f37d89ab5ff74ef49387eca385b85c66 (127.31.204.195:39711): tablet 36305bef78eb4cf28f6f2ed951dbf251 (table test-table [id=4eec9162554f4a2598fe1c59bd19f898]) successfully deleted
I20260810 06:20:30.036201 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 7317
I20260810 06:20:30.044114 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 6723
2026-08-10T06:20:30Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2382 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T06:20:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:20:30Z Disabled control of system clock
I20260810 06:20:30.074915 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:35887
--webserver_interface=127.31.204.254
--webserver_port=0
--builtin_ntp_servers=127.31.204.212:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:35887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 06:20:30.179994 7619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:30.180229 7619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:30.180285 7619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:30.184176 7619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:30.184232 7619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:30.184258 7619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:30.184285 7619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:30.184319 7619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:20:30.189003 7619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:35887
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:35887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:30.189882 7619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:30.191123 7619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:30.198859 7628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:30.198922 7627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:30.198909 7619 server_base.cc:1034] running on GCE node
W20260810 06:20:30.199012 7633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:30.199579 7619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:30.200512 7619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:30.201675 7619 hybrid_clock.cc:648] HybridClock initialized: now 1786342830201657 us; error 31 us; skew 500 ppm
I20260810 06:20:30.203305 7619 webserver.cc:460] Webserver started at http://127.31.204.254:36567/ using document root <none> and password file <none>
I20260810 06:20:30.203789 7619 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:30.203847 7619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:30.204030 7619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:30.205834 7619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "6b222cab2dd34e259c7c1b1f0fd39c52"
format_stamp: "Formatted at 2026-08-10 06:20:30 on dist-test-slave-q9h9"
server_key: "3d6a32e513ad00bdf1d09d252b13b2d7"
server_key_iv: "10a3c16fde64bdd47843ac014c219924"
server_key_version: "encryptionkey@0"
I20260810 06:20:30.206367 7619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "6b222cab2dd34e259c7c1b1f0fd39c52"
format_stamp: "Formatted at 2026-08-10 06:20:30 on dist-test-slave-q9h9"
server_key: "3d6a32e513ad00bdf1d09d252b13b2d7"
server_key_iv: "10a3c16fde64bdd47843ac014c219924"
server_key_version: "encryptionkey@0"
I20260810 06:20:30.209565 7619 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:20:30.211767 7642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:30.212839 7619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:30.212934 7619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "6b222cab2dd34e259c7c1b1f0fd39c52"
format_stamp: "Formatted at 2026-08-10 06:20:30 on dist-test-slave-q9h9"
server_key: "3d6a32e513ad00bdf1d09d252b13b2d7"
server_key_iv: "10a3c16fde64bdd47843ac014c219924"
server_key_version: "encryptionkey@0"
I20260810 06:20:30.213028 7619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:30.217137 7619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:30.217646 7619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:30.217803 7619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:30.224179 7619 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:35887
I20260810 06:20:30.224205 7732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:35887 every 8 connection(s)
I20260810 06:20:30.224952 7619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:20:30.227926 7734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:30.229589 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 7619
I20260810 06:20:30.229703 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:20:30.229992 32563 external_mini_cluster.cc:1428] Setting key 174018cf39872a97dbfab70f013998fd
I20260810 06:20:30.234465 7734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52: Bootstrap starting.
I20260810 06:20:30.235442 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.231298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.31.204.254:35887)
0810 06:20:30.231914 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:30.231934 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:30.231975 (+ 41us) server_negotiation.cc:407] Connection header received
0810 06:20:30.232657 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:30.232679 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:30.233087 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:30.233444 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:30.234083 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:30.234088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:30.234102 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:20:30.234113 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:30.234182 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:20:30.234203 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:30.234333 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:30.234397 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 06:20:30.234620 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":94,"threads_started":1}
I20260810 06:20:30.236694 7734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:30.237353 7734 log.cc:824] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:30.239029 7734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52: No bootstrap required, opened a new log
I20260810 06:20:30.242231 7734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b222cab2dd34e259c7c1b1f0fd39c52" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35887 } }
I20260810 06:20:30.242419 7734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:30.242475 7734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b222cab2dd34e259c7c1b1f0fd39c52, State: Initialized, Role: FOLLOWER
I20260810 06:20:30.242865 7734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b222cab2dd34e259c7c1b1f0fd39c52" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35887 } }
I20260810 06:20:30.243000 7734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:20:30.243064 7734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:20:30.243153 7734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:30.243911 7734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b222cab2dd34e259c7c1b1f0fd39c52" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35887 } }
I20260810 06:20:30.244212 7734 leader_election.cc:302] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b222cab2dd34e259c7c1b1f0fd39c52; no voters:
I20260810 06:20:30.244459 7734 leader_election.cc:288] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:20:30.244572 7741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:30.244786 7741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52 [term 1 LEADER]: Becoming Leader. State: Replica: 6b222cab2dd34e259c7c1b1f0fd39c52, State: Running, Role: LEADER
I20260810 06:20:30.245052 7741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b222cab2dd34e259c7c1b1f0fd39c52" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35887 } }
I20260810 06:20:30.245481 7734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:30.246532 7742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b222cab2dd34e259c7c1b1f0fd39c52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b222cab2dd34e259c7c1b1f0fd39c52" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35887 } } }
I20260810 06:20:30.246560 7743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b222cab2dd34e259c7c1b1f0fd39c52. Latest consensus state: current_term: 1 leader_uuid: "6b222cab2dd34e259c7c1b1f0fd39c52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b222cab2dd34e259c7c1b1f0fd39c52" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35887 } } }
I20260810 06:20:30.246758 7743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52 [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:30.246825 7742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52 [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:30.247267 7755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:30.249974 7755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:30.254899 7755 catalog_manager.cc:1347] Generated new cluster ID: 7ad34ae7137243e6a7c7900635dcbbac
I20260810 06:20:30.254951 7755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:30.277133 7755 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:20:30.277408 7755 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:30.288735 7755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52: Generated new TSK 0
I20260810 06:20:30.289326 7755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:20:30.293977 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35887
--builtin_ntp_servers=127.31.204.212:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 06:20:30.401547 7771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:30.401762 7771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:30.401798 7771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:30.405642 7771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:30.405685 7771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:30.405736 7771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:20:30.410233 7771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:30.411034 7771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:30.412171 7771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:30.420877 7782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:30.420991 7771 server_base.cc:1034] running on GCE node
W20260810 06:20:30.420902 7789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:30.421159 7784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:30.421557 7771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:30.422305 7771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:30.423480 7771 hybrid_clock.cc:648] HybridClock initialized: now 1786342830423460 us; error 40 us; skew 500 ppm
I20260810 06:20:30.425062 7771 webserver.cc:460] Webserver started at http://127.31.204.193:45983/ using document root <none> and password file <none>
I20260810 06:20:30.425542 7771 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:30.425597 7771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:30.425750 7771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:30.427549 7771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "ca4f2842f8c44f3f84d8ef4f745a8f42"
format_stamp: "Formatted at 2026-08-10 06:20:30 on dist-test-slave-q9h9"
server_key: "ff9776bf7759dfd9f50fd644261d9a45"
server_key_iv: "b35ef8f2a6d9f890f08c76e0bfa920f6"
server_key_version: "encryptionkey@0"
I20260810 06:20:30.428009 7771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "ca4f2842f8c44f3f84d8ef4f745a8f42"
format_stamp: "Formatted at 2026-08-10 06:20:30 on dist-test-slave-q9h9"
server_key: "ff9776bf7759dfd9f50fd644261d9a45"
server_key_iv: "b35ef8f2a6d9f890f08c76e0bfa920f6"
server_key_version: "encryptionkey@0"
I20260810 06:20:30.431250 7771 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:20:30.433357 7803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:30.434448 7771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:30.434546 7771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "ca4f2842f8c44f3f84d8ef4f745a8f42"
format_stamp: "Formatted at 2026-08-10 06:20:30 on dist-test-slave-q9h9"
server_key: "ff9776bf7759dfd9f50fd644261d9a45"
server_key_iv: "b35ef8f2a6d9f890f08c76e0bfa920f6"
server_key_version: "encryptionkey@0"
I20260810 06:20:30.434635 7771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:30.438725 7771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:30.439239 7771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:30.439393 7771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:30.439854 7771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:30.440634 7771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:30.440678 7771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:30.440725 7771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:30.440754 7771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:30.448767 7771 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:46547
I20260810 06:20:30.448824 8014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:46547 every 8 connection(s)
I20260810 06:20:30.449515 7771 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:20:30.450037 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 7771
I20260810 06:20:30.450165 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:20:30.450452 32563 external_mini_cluster.cc:1428] Setting key d5bd5c955d73f5f3df25fc6e0c37b06f
I20260810 06:20:30.452466 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35887
--builtin_ntp_servers=127.31.204.212:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 06:20:30.455555 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.451668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:42393 (local address 127.31.204.254:35887)
0810 06:20:30.451782 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:30.451785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:30.452506 (+ 721us) server_negotiation.cc:407] Connection header received
0810 06:20:30.453961 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:30.453964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:30.453999 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:30.454094 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:30.455154 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:30.455158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:30.455161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:20:30.455175 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:30.455199 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:20:30.455210 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:30.455318 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:30.455408 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 06:20:30.455449 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:20:30.456249 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.451594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:35887 (local address 127.31.204.193:42393)
0810 06:20:30.452352 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:30.452403 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:30.453760 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:30.454273 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:30.454281 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:30.454389 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:30.454911 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:20:30.454912 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:30.454948 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:20:30.455009 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:30.455300 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:30.455304 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:30.455307 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 06:20:30.455418 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 06:20:30.455584 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":139,"threads_started":1}
I20260810 06:20:30.457473 8015 heartbeater.cc:347] Connected to a master server at 127.31.204.254:35887
I20260810 06:20:30.457665 8015 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:30.458039 8015 heartbeater.cc:511] Master 127.31.204.254:35887 requested a full tablet report, sending...
I20260810 06:20:30.459609 7673 ts_manager.cc:194] Registered new tserver with Master: ca4f2842f8c44f3f84d8ef4f745a8f42 (127.31.204.193:46547)
W20260810 06:20:30.557039 8019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:30.557231 8019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:30.557273 8019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:30.561007 8019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:30.561046 8019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:30.561094 8019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:20:30.565455 8019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:30.566215 8019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:30.567299 8019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:30.575552 8028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:30.575675 8026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:30.575730 8019 server_base.cc:1034] running on GCE node
W20260810 06:20:30.575675 8032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:30.576205 8019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:30.576730 8019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:30.577884 8019 hybrid_clock.cc:648] HybridClock initialized: now 1786342830577865 us; error 31 us; skew 500 ppm
I20260810 06:20:30.579437 8019 webserver.cc:460] Webserver started at http://127.31.204.194:38271/ using document root <none> and password file <none>
I20260810 06:20:30.579906 8019 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:30.579963 8019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:30.580142 8019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:30.581871 8019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "c426348ef99e4281be56821877fb96de"
format_stamp: "Formatted at 2026-08-10 06:20:30 on dist-test-slave-q9h9"
server_key: "05847d2a27a0cc7161acc74493f115fb"
server_key_iv: "e2e2031cee602996457ad8e0907243bd"
server_key_version: "encryptionkey@0"
I20260810 06:20:30.582399 8019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "c426348ef99e4281be56821877fb96de"
format_stamp: "Formatted at 2026-08-10 06:20:30 on dist-test-slave-q9h9"
server_key: "05847d2a27a0cc7161acc74493f115fb"
server_key_iv: "e2e2031cee602996457ad8e0907243bd"
server_key_version: "encryptionkey@0"
I20260810 06:20:30.585505 8019 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:20:30.587610 8040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:30.588626 8019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:30.588723 8019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "c426348ef99e4281be56821877fb96de"
format_stamp: "Formatted at 2026-08-10 06:20:30 on dist-test-slave-q9h9"
server_key: "05847d2a27a0cc7161acc74493f115fb"
server_key_iv: "e2e2031cee602996457ad8e0907243bd"
server_key_version: "encryptionkey@0"
I20260810 06:20:30.588815 8019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:30.592790 8019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:30.593345 8019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:30.593492 8019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:30.593946 8019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:30.594729 8019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:30.594776 8019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:30.594843 8019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:30.594874 8019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:30.602726 8019 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:33795
I20260810 06:20:30.602769 8239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:33795 every 8 connection(s)
I20260810 06:20:30.603482 8019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:20:30.609216 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 8019
I20260810 06:20:30.609359 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:20:30.609688 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.605128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:43443 (local address 127.31.204.254:35887)
0810 06:20:30.605261 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:30.605265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:30.606387 (+ 1122us) server_negotiation.cc:407] Connection header received
0810 06:20:30.607707 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:30.607710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:30.607763 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:30.607844 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:30.609077 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:30.609081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:30.609083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:20:30.609095 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:30.609132 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:20:30.609144 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:30.609240 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:30.609529 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 06:20:30.609576 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:20:30.609730 32563 external_mini_cluster.cc:1428] Setting key 2fae57000d8ae65b4b86ed6eb9db3fd1
I20260810 06:20:30.610678 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.605426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:35887 (local address 127.31.204.194:43443)
0810 06:20:30.606226 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:30.606267 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:30.607485 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:30.608027 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:30.608033 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:30.608152 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:30.608783 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:20:30.608785 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:30.608825 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:20:30.608901 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:30.609286 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:30.609297 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:30.609305 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 06:20:30.609558 (+ 253us) client_negotiation.cc:240] Negotiation successful
0810 06:20:30.609800 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":115,"threads_started":1}
I20260810 06:20:30.611745 8240 heartbeater.cc:347] Connected to a master server at 127.31.204.254:35887
I20260810 06:20:30.611887 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35887
--builtin_ntp_servers=127.31.204.212:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 06:20:30.611958 8240 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:30.612356 8240 heartbeater.cc:511] Master 127.31.204.254:35887 requested a full tablet report, sending...
I20260810 06:20:30.613277 7673 ts_manager.cc:194] Registered new tserver with Master: c426348ef99e4281be56821877fb96de (127.31.204.194:33795)
W20260810 06:20:30.716876 8246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:30.717077 8246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:30.717129 8246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:30.721004 8246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:30.721048 8246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:30.721118 8246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:20:30.725570 8246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:30.726402 8246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:30.727557 8246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:30.735914 8256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:30.735898 8259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:30.735877 8257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:30.736136 8246 server_base.cc:1034] running on GCE node
I20260810 06:20:30.736487 8246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:30.737002 8246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:30.738163 8246 hybrid_clock.cc:648] HybridClock initialized: now 1786342830738143 us; error 35 us; skew 500 ppm
I20260810 06:20:30.739730 8246 webserver.cc:460] Webserver started at http://127.31.204.195:46559/ using document root <none> and password file <none>
I20260810 06:20:30.740191 8246 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:30.740252 8246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:30.740442 8246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:30.742249 8246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "b75ac312b586472a8ca8534a84b6a86c"
format_stamp: "Formatted at 2026-08-10 06:20:30 on dist-test-slave-q9h9"
server_key: "b88054de8625a8d3474180ef11070a3c"
server_key_iv: "116a4ada56d1f390738dd9a17039cf2b"
server_key_version: "encryptionkey@0"
I20260810 06:20:30.742802 8246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "b75ac312b586472a8ca8534a84b6a86c"
format_stamp: "Formatted at 2026-08-10 06:20:30 on dist-test-slave-q9h9"
server_key: "b88054de8625a8d3474180ef11070a3c"
server_key_iv: "116a4ada56d1f390738dd9a17039cf2b"
server_key_version: "encryptionkey@0"
I20260810 06:20:30.746166 8246 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:20:30.748461 8269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:30.749473 8246 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:30.749558 8246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "b75ac312b586472a8ca8534a84b6a86c"
format_stamp: "Formatted at 2026-08-10 06:20:30 on dist-test-slave-q9h9"
server_key: "b88054de8625a8d3474180ef11070a3c"
server_key_iv: "116a4ada56d1f390738dd9a17039cf2b"
server_key_version: "encryptionkey@0"
I20260810 06:20:30.749652 8246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:30.753636 8246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:30.754194 8246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:30.754343 8246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:30.754793 8246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:30.755589 8246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:30.755632 8246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:30.755669 8246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:30.755698 8246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:30.763329 8246 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:37795
I20260810 06:20:30.763383 8463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:37795 every 8 connection(s)
I20260810 06:20:30.764047 8246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:20:30.767910 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 8246
I20260810 06:20:30.768018 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:20:30.768244 32563 external_mini_cluster.cc:1428] Setting key 92aa7ef4ac0f82f96d6baac53b2d2016
I20260810 06:20:30.770009 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.765576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:56149 (local address 127.31.204.254:35887)
0810 06:20:30.765705 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:30.765708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:30.766655 (+ 947us) server_negotiation.cc:407] Connection header received
0810 06:20:30.768524 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:30.768527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:30.768564 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:30.768640 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:30.769594 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:30.769597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:30.769600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:20:30.769613 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:30.769633 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:20:30.769641 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:30.769712 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:30.769845 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 06:20:30.769887 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:20:30.770766 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.765880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:35887 (local address 127.31.204.195:56149)
0810 06:20:30.766508 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:30.766542 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:30.767595 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:30.768804 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:30.768810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:30.768887 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:30.769358 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:20:30.769360 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:30.769395 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:20:30.769455 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:30.769714 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:30.769723 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:30.769729 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 06:20:30.769874 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 06:20:30.770107 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":151,"threads_started":1}
I20260810 06:20:30.771697 8464 heartbeater.cc:347] Connected to a master server at 127.31.204.254:35887
I20260810 06:20:30.771878 8464 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:30.772188 8464 heartbeater.cc:511] Master 127.31.204.254:35887 requested a full tablet report, sending...
I20260810 06:20:30.773043 7673 ts_manager.cc:194] Registered new tserver with Master: b75ac312b586472a8ca8534a84b6a86c (127.31.204.195:37795)
I20260810 06:20:30.781610 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:20:30.784740 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.783339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39122 (local address 127.31.204.254:35887)
0810 06:20:30.783448 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:30.783452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:30.783628 (+ 176us) server_negotiation.cc:407] Connection header received
0810 06:20:30.783728 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:30.783731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:30.783772 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:30.783832 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:30.784406 (+ 574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:30.784409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:30.784412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:20:30.784423 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:30.784440 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:20:30.784448 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:30.784515 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:30.784598 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 06:20:30.784634 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 06:20:30.787213 7673 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:20:30.790007 7673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:20:30.800387 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.797505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:37795 (local address 127.0.0.1:34256)
0810 06:20:30.797790 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:30.797835 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:30.797923 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:30.798722 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:30.798727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:30.798792 (+ 65us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:30.799245 (+ 453us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:20:30.799248 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:30.799281 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:20:30.799333 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:30.800090 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:30.800100 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:30.800105 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 06:20:30.800205 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 06:20:30.800273 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":57,"threads_started":1}
I20260810 06:20:30.800421 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.798054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.31.204.195:37795)
0810 06:20:30.798324 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:30.798336 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:30.798358 (+ 22us) server_negotiation.cc:407] Connection header received
0810 06:20:30.798399 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:30.798404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:30.798568 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:30.799028 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:30.799910 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:30.799916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:30.799927 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:20:30.799937 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:30.799972 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:20:30.799997 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:30.800090 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:30.800230 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 06:20:30.800305 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":90,"threads_started":1}
I20260810 06:20:30.800653 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.797653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:33795 (local address 127.0.0.1:33810)
0810 06:20:30.797924 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:30.797937 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:30.798034 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:30.799102 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:30.799105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:30.799120 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:30.799485 (+ 365us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:20:30.799487 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:30.799497 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:20:30.799532 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:30.800419 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:30.800422 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:30.800425 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 06:20:30.800500 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 06:20:30.800542 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":100,"threads_started":1}
I20260810 06:20:30.801025 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.798176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33810 (local address 127.31.204.194:33795)
0810 06:20:30.798484 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:30.798496 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:30.798519 (+ 23us) server_negotiation.cc:407] Connection header received
0810 06:20:30.798571 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:30.798576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:30.798737 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:30.798868 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:30.800179 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:30.800202 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:30.800213 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:20:30.800229 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:30.800275 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:20:30.800309 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:30.800417 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:30.800827 (+ 410us) server_negotiation.cc:299] Negotiation successful
0810 06:20:30.800890 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":115,"threads_started":1}
I20260810 06:20:30.801448 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.798257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:46547 (local address 127.0.0.1:40794)
0810 06:20:30.799193 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:30.799206 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:30.799285 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:30.799820 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:30.799823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:30.799836 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:30.800718 (+ 882us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:20:30.800719 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:30.800832 (+ 113us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:20:30.800858 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:30.801232 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:30.801236 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:30.801239 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 06:20:30.801308 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 06:20:30.801340 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"thread_start_us":78,"threads_started":1}
I20260810 06:20:30.801916 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.798641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40794 (local address 127.31.204.193:46547)
0810 06:20:30.799060 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:30.799069 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:30.799379 (+ 310us) server_negotiation.cc:407] Connection header received
0810 06:20:30.799440 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:30.799462 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:30.799666 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:30.799824 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:30.800979 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:30.800984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:30.800994 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:20:30.801005 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:30.801046 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:20:30.801075 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:30.801175 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:30.801704 (+ 529us) server_negotiation.cc:299] Negotiation successful
0810 06:20:30.801773 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":133,"threads_started":1}
I20260810 06:20:30.803076 8353 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb87a3708b847d7ad2c3d61607fce6d (DEFAULT_TABLE table=test-table [id=f7ecfaa0f848477ea2ca0655ee859b04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:30.803417 8125 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb87a3708b847d7ad2c3d61607fce6d (DEFAULT_TABLE table=test-table [id=f7ecfaa0f848477ea2ca0655ee859b04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:30.804097 8353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb87a3708b847d7ad2c3d61607fce6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:30.804314 8125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb87a3708b847d7ad2c3d61607fce6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:30.804404 7896 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb87a3708b847d7ad2c3d61607fce6d (DEFAULT_TABLE table=test-table [id=f7ecfaa0f848477ea2ca0655ee859b04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:30.805327 7896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb87a3708b847d7ad2c3d61607fce6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:30.809636 8496 tablet_bootstrap.cc:492] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de: Bootstrap starting.
I20260810 06:20:30.811172 8497 tablet_bootstrap.cc:492] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c: Bootstrap starting.
I20260810 06:20:30.811657 8498 tablet_bootstrap.cc:492] T 8cb87a3708b847d7ad2c3d61607fce6d P ca4f2842f8c44f3f84d8ef4f745a8f42: Bootstrap starting.
I20260810 06:20:30.811864 8496 tablet_bootstrap.cc:654] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:30.812754 8496 log.cc:824] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:30.813174 8497 tablet_bootstrap.cc:654] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:30.813648 8498 tablet_bootstrap.cc:654] T 8cb87a3708b847d7ad2c3d61607fce6d P ca4f2842f8c44f3f84d8ef4f745a8f42: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:30.813922 8497 log.cc:824] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:30.814355 8498 log.cc:824] T 8cb87a3708b847d7ad2c3d61607fce6d P ca4f2842f8c44f3f84d8ef4f745a8f42: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:30.814842 8496 tablet_bootstrap.cc:492] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de: No bootstrap required, opened a new log
I20260810 06:20:30.815032 8496 ts_tablet_manager.cc:1397] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:20:30.815817 8497 tablet_bootstrap.cc:492] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c: No bootstrap required, opened a new log
I20260810 06:20:30.815958 8498 tablet_bootstrap.cc:492] T 8cb87a3708b847d7ad2c3d61607fce6d P ca4f2842f8c44f3f84d8ef4f745a8f42: No bootstrap required, opened a new log
I20260810 06:20:30.816009 8497 ts_tablet_manager.cc:1397] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 06:20:30.816184 8498 ts_tablet_manager.cc:1397] T 8cb87a3708b847d7ad2c3d61607fce6d P ca4f2842f8c44f3f84d8ef4f745a8f42: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:20:30.819357 8496 raft_consensus.cc:348] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75ac312b586472a8ca8534a84b6a86c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37795 } } peers { permanent_uuid: "c426348ef99e4281be56821877fb96de" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 33795 } } peers { permanent_uuid: "ca4f2842f8c44f3f84d8ef4f745a8f42" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 46547 } }
I20260810 06:20:30.819604 8496 raft_consensus.cc:374] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:30.819667 8496 raft_consensus.cc:729] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c426348ef99e4281be56821877fb96de, State: Initialized, Role: FOLLOWER
I20260810 06:20:30.819554 8498 raft_consensus.cc:348] T 8cb87a3708b847d7ad2c3d61607fce6d P ca4f2842f8c44f3f84d8ef4f745a8f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75ac312b586472a8ca8534a84b6a86c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37795 } } peers { permanent_uuid: "c426348ef99e4281be56821877fb96de" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 33795 } } peers { permanent_uuid: "ca4f2842f8c44f3f84d8ef4f745a8f42" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 46547 } }
I20260810 06:20:30.819778 8498 raft_consensus.cc:374] T 8cb87a3708b847d7ad2c3d61607fce6d P ca4f2842f8c44f3f84d8ef4f745a8f42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:30.819845 8498 raft_consensus.cc:729] T 8cb87a3708b847d7ad2c3d61607fce6d P ca4f2842f8c44f3f84d8ef4f745a8f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca4f2842f8c44f3f84d8ef4f745a8f42, State: Initialized, Role: FOLLOWER
I20260810 06:20:30.820113 8496 consensus_queue.cc:260] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75ac312b586472a8ca8534a84b6a86c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37795 } } peers { permanent_uuid: "c426348ef99e4281be56821877fb96de" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 33795 } } peers { permanent_uuid: "ca4f2842f8c44f3f84d8ef4f745a8f42" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 46547 } }
I20260810 06:20:30.820211 8498 consensus_queue.cc:260] T 8cb87a3708b847d7ad2c3d61607fce6d P ca4f2842f8c44f3f84d8ef4f745a8f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75ac312b586472a8ca8534a84b6a86c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37795 } } peers { permanent_uuid: "c426348ef99e4281be56821877fb96de" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 33795 } } peers { permanent_uuid: "ca4f2842f8c44f3f84d8ef4f745a8f42" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 46547 } }
I20260810 06:20:30.820169 8497 raft_consensus.cc:348] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75ac312b586472a8ca8534a84b6a86c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37795 } } peers { permanent_uuid: "c426348ef99e4281be56821877fb96de" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 33795 } } peers { permanent_uuid: "ca4f2842f8c44f3f84d8ef4f745a8f42" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 46547 } }
I20260810 06:20:30.820384 8497 raft_consensus.cc:374] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:30.820438 8497 raft_consensus.cc:729] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75ac312b586472a8ca8534a84b6a86c, State: Initialized, Role: FOLLOWER
I20260810 06:20:30.820855 8497 consensus_queue.cc:260] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75ac312b586472a8ca8534a84b6a86c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37795 } } peers { permanent_uuid: "c426348ef99e4281be56821877fb96de" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 33795 } } peers { permanent_uuid: "ca4f2842f8c44f3f84d8ef4f745a8f42" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 46547 } }
I20260810 06:20:30.821106 8496 ts_tablet_manager.cc:1428] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260810 06:20:30.821115 8498 ts_tablet_manager.cc:1428] T 8cb87a3708b847d7ad2c3d61607fce6d P ca4f2842f8c44f3f84d8ef4f745a8f42: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:20:30.821416 8240 heartbeater.cc:503] Master 127.31.204.254:35887 was elected leader, sending a full tablet report...
I20260810 06:20:30.821424 8015 heartbeater.cc:503] Master 127.31.204.254:35887 was elected leader, sending a full tablet report...
I20260810 06:20:30.821671 8497 ts_tablet_manager.cc:1428] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 06:20:30.821961 8464 heartbeater.cc:503] Master 127.31.204.254:35887 was elected leader, sending a full tablet report...
W20260810 06:20:30.854316 8243 tablet.cc:2369] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:20:30.950634 8016 tablet.cc:2369] T 8cb87a3708b847d7ad2c3d61607fce6d P ca4f2842f8c44f3f84d8ef4f745a8f42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:20:30.968963 8504 raft_consensus.cc:482] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:20:30.969105 8504 raft_consensus.cc:504] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75ac312b586472a8ca8534a84b6a86c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37795 } } peers { permanent_uuid: "c426348ef99e4281be56821877fb96de" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 33795 } } peers { permanent_uuid: "ca4f2842f8c44f3f84d8ef4f745a8f42" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 46547 } }
I20260810 06:20:30.970118 8504 leader_election.cc:288] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b75ac312b586472a8ca8534a84b6a86c (127.31.204.195:37795), ca4f2842f8c44f3f84d8ef4f745a8f42 (127.31.204.193:46547)
I20260810 06:20:30.971782 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.970303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:37795 (local address 127.31.204.194:45331)
0810 06:20:30.970420 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:30.970434 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:30.970509 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:30.970749 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:30.970752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:30.970767 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:30.971213 (+ 446us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:20:30.971215 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:30.971227 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:20:30.971256 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:30.971535 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:30.971538 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:30.971542 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 06:20:30.971624 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 06:20:30.971660 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 06:20:30.971788 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.970394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:45331 (local address 127.31.204.195:37795)
0810 06:20:30.970528 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:30.970533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:30.970551 (+ 18us) server_negotiation.cc:407] Connection header received
0810 06:20:30.970592 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:30.970595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:30.970634 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:30.970728 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:30.971392 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:30.971396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:30.971398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:20:30.971412 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:30.971434 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:20:30.971445 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:30.971538 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:30.971628 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 06:20:30.971664 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 06:20:30.971935 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.970546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:38229 (local address 127.31.204.193:46547)
0810 06:20:30.970665 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:30.970669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:30.970833 (+ 164us) server_negotiation.cc:407] Connection header received
0810 06:20:30.970913 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:30.970916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:30.970947 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:30.971005 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:30.971572 (+ 567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:30.971576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:30.971579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:20:30.971592 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:30.971612 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:20:30.971622 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:30.971709 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:30.971790 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 06:20:30.971827 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 06:20:30.971956 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.970361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:46547 (local address 127.31.204.194:38229)
0810 06:20:30.970733 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:30.970743 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:30.970832 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:30.971044 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:30.971047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:30.971061 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:30.971419 (+ 358us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:20:30.971420 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:30.971430 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:20:30.971452 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:30.971700 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:30.971703 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:30.971706 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 06:20:30.971791 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 06:20:30.971826 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":124,"threads_started":1}
I20260810 06:20:30.972504 8388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb87a3708b847d7ad2c3d61607fce6d" candidate_uuid: "c426348ef99e4281be56821877fb96de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b75ac312b586472a8ca8534a84b6a86c" is_pre_election: true
I20260810 06:20:30.972534 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb87a3708b847d7ad2c3d61607fce6d" candidate_uuid: "c426348ef99e4281be56821877fb96de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca4f2842f8c44f3f84d8ef4f745a8f42" is_pre_election: true
I20260810 06:20:30.972780 8388 raft_consensus.cc:2454] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c426348ef99e4281be56821877fb96de in term 0.
I20260810 06:20:30.972784 7926 raft_consensus.cc:2454] T 8cb87a3708b847d7ad2c3d61607fce6d P ca4f2842f8c44f3f84d8ef4f745a8f42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c426348ef99e4281be56821877fb96de in term 0.
I20260810 06:20:30.973222 8043 leader_election.cc:302] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b75ac312b586472a8ca8534a84b6a86c, c426348ef99e4281be56821877fb96de; no voters:
I20260810 06:20:30.973448 8504 raft_consensus.cc:2781] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:20:30.973534 8504 raft_consensus.cc:482] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:20:30.973584 8504 raft_consensus.cc:3037] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:30.974581 8504 raft_consensus.cc:504] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75ac312b586472a8ca8534a84b6a86c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37795 } } peers { permanent_uuid: "c426348ef99e4281be56821877fb96de" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 33795 } } peers { permanent_uuid: "ca4f2842f8c44f3f84d8ef4f745a8f42" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 46547 } }
I20260810 06:20:30.974887 8504 leader_election.cc:288] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [CANDIDATE]: Term 1 election: Requested vote from peers b75ac312b586472a8ca8534a84b6a86c (127.31.204.195:37795), ca4f2842f8c44f3f84d8ef4f745a8f42 (127.31.204.193:46547)
I20260810 06:20:30.975200 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb87a3708b847d7ad2c3d61607fce6d" candidate_uuid: "c426348ef99e4281be56821877fb96de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca4f2842f8c44f3f84d8ef4f745a8f42"
I20260810 06:20:30.975222 8388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb87a3708b847d7ad2c3d61607fce6d" candidate_uuid: "c426348ef99e4281be56821877fb96de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b75ac312b586472a8ca8534a84b6a86c"
I20260810 06:20:30.975320 8388 raft_consensus.cc:3037] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:30.975322 7926 raft_consensus.cc:3037] T 8cb87a3708b847d7ad2c3d61607fce6d P ca4f2842f8c44f3f84d8ef4f745a8f42 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:30.976532 7926 raft_consensus.cc:2454] T 8cb87a3708b847d7ad2c3d61607fce6d P ca4f2842f8c44f3f84d8ef4f745a8f42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c426348ef99e4281be56821877fb96de in term 1.
I20260810 06:20:30.976532 8388 raft_consensus.cc:2454] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c426348ef99e4281be56821877fb96de in term 1.
I20260810 06:20:30.976796 8044 leader_election.cc:302] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c426348ef99e4281be56821877fb96de, ca4f2842f8c44f3f84d8ef4f745a8f42; no voters:
I20260810 06:20:30.976953 8504 raft_consensus.cc:2781] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:30.977133 8504 raft_consensus.cc:686] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [term 1 LEADER]: Becoming Leader. State: Replica: c426348ef99e4281be56821877fb96de, State: Running, Role: LEADER
I20260810 06:20:30.977412 8504 consensus_queue.cc:237] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75ac312b586472a8ca8534a84b6a86c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37795 } } peers { permanent_uuid: "c426348ef99e4281be56821877fb96de" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 33795 } } peers { permanent_uuid: "ca4f2842f8c44f3f84d8ef4f745a8f42" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 46547 } }
I20260810 06:20:30.980207 7672 catalog_manager.cc:5495] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de reported cstate change: term changed from 0 to 1, leader changed from <none> to c426348ef99e4281be56821877fb96de (127.31.204.194). New cstate: current_term: 1 leader_uuid: "c426348ef99e4281be56821877fb96de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75ac312b586472a8ca8534a84b6a86c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c426348ef99e4281be56821877fb96de" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 33795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca4f2842f8c44f3f84d8ef4f745a8f42" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 46547 } health_report { overall_health: UNKNOWN } } }
I20260810 06:20:30.992648 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:30.991345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33812 (local address 127.31.204.194:33795)
0810 06:20:30.991453 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:30.991456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:30.991468 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:20:30.991606 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:30.991609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:30.991673 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:30.991743 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:30.992313 (+ 570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:30.992317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:30.992319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:20:30.992330 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:30.992347 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:20:30.992356 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:30.992427 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:30.992513 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 06:20:30.992545 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:20:30.997529 8388 raft_consensus.cc:1261] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c [term 1 FOLLOWER]: Refusing update from remote peer c426348ef99e4281be56821877fb96de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:30.997529 7926 raft_consensus.cc:1261] T 8cb87a3708b847d7ad2c3d61607fce6d P ca4f2842f8c44f3f84d8ef4f745a8f42 [term 1 FOLLOWER]: Refusing update from remote peer c426348ef99e4281be56821877fb96de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:30.998203 8510 consensus_queue.cc:1035] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [LEADER]: Connected to new peer: Peer: permanent_uuid: "b75ac312b586472a8ca8534a84b6a86c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 37795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:30.998364 8504 consensus_queue.cc:1035] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca4f2842f8c44f3f84d8ef4f745a8f42" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:31.012858 7672 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:31.012988 7672 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:31.015198 7672 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b222cab2dd34e259c7c1b1f0fd39c52: Sending DeleteTablet for 3 replicas of tablet 8cb87a3708b847d7ad2c3d61607fce6d
I20260810 06:20:31.015877 8353 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cb87a3708b847d7ad2c3d61607fce6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:31 UTC) from {username='slave'} at 127.0.0.1:34256
I20260810 06:20:31.015890 8125 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cb87a3708b847d7ad2c3d61607fce6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:31 UTC) from {username='slave'} at 127.0.0.1:33810
I20260810 06:20:31.016023 7896 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cb87a3708b847d7ad2c3d61607fce6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:31 UTC) from {username='slave'} at 127.0.0.1:40794
I20260810 06:20:31.016386 8540 tablet_replica.cc:330] T 8cb87a3708b847d7ad2c3d61607fce6d P ca4f2842f8c44f3f84d8ef4f745a8f42: stopping tablet replica
I20260810 06:20:31.016503 8538 tablet_replica.cc:330] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c: stopping tablet replica
I20260810 06:20:31.016567 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 7771
I20260810 06:20:31.016772 8538 raft_consensus.cc:2229] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:31.017077 8538 raft_consensus.cc:2258] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:31.025293 8538 ts_tablet_manager.cc:1905] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:31.028466 8538 ts_tablet_manager.cc:1918] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:20:31.028573 8538 log.cc:1196] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/8cb87a3708b847d7ad2c3d61607fce6d
I20260810 06:20:31.028970 8538 ts_tablet_manager.cc:1939] T 8cb87a3708b847d7ad2c3d61607fce6d P b75ac312b586472a8ca8534a84b6a86c: Deleting consensus metadata
I20260810 06:20:31.016548 8539 tablet_replica.cc:330] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de: stopping tablet replica
I20260810 06:20:31.034757 7645 catalog_manager.cc:4841] TS b75ac312b586472a8ca8534a84b6a86c (127.31.204.195:37795): tablet 8cb87a3708b847d7ad2c3d61607fce6d (table test-table [id=f7ecfaa0f848477ea2ca0655ee859b04]) successfully deleted
I20260810 06:20:31.035085 8539 raft_consensus.cc:2229] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:20:31.035432 8539 raft_consensus.cc:2258] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:31.036563 8539 ts_tablet_manager.cc:1905] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:31.038679 8539 ts_tablet_manager.cc:1918] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:20:31.038764 8539 log.cc:1196] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/8cb87a3708b847d7ad2c3d61607fce6d
I20260810 06:20:31.039031 8539 ts_tablet_manager.cc:1939] T 8cb87a3708b847d7ad2c3d61607fce6d P c426348ef99e4281be56821877fb96de: Deleting consensus metadata
I20260810 06:20:31.039940 7644 catalog_manager.cc:4841] TS c426348ef99e4281be56821877fb96de (127.31.204.194:33795): tablet 8cb87a3708b847d7ad2c3d61607fce6d (table test-table [id=f7ecfaa0f848477ea2ca0655ee859b04]) successfully deleted
W20260810 06:20:31.041501 7646 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.204.193:46547 (error 108)
I20260810 06:20:31.042322 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 8019
I20260810 06:20:31.042456 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:31.042185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:46547 (local address 127.0.0.1:40808)
0810 06:20:31.042312 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:31.042383 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 06:20:31.045679 7646 catalog_manager.cc:4584] TS ca4f2842f8c44f3f84d8ef4f745a8f42 (127.31.204.193:46547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8cb87a3708b847d7ad2c3d61607fce6d: Network error: Client connection negotiation failed: client connection to 127.31.204.193:46547: connect: Connection refused (error 111)
I20260810 06:20:31.063532 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 8246
I20260810 06:20:31.076628 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 7619
2026-08-10T06:20:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1034 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 06:20:31.101697 32563 mini_oidc.cc:152] Starting JWKS server
I20260810 06:20:31.103506 32563 webserver.cc:460] Webserver started at https://127.0.0.1:42935/ using document root <none> and password file <none>
I20260810 06:20:31.103659 32563 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 06:20:31.103953 32563 webserver.cc:460] Webserver started at http://127.0.0.1:44993/ using document root <none> and password file <none>
2026-08-10T06:20:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:20:31Z Disabled control of system clock
I20260810 06:20:31.112905 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:34367
--webserver_interface=127.31.204.254
--webserver_port=0
--builtin_ntp_servers=127.31.204.212:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:34367
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42935/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0/testchainca.cert with env {}
W20260810 06:20:31.218257 8554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:31.218492 8554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:31.218545 8554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:31.222445 8554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:31.222494 8554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:31.222515 8554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:31.222537 8554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:31.222565 8554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 06:20:31.222589 8554 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 06:20:31.222615 8554 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42935/jwks.json
I20260810 06:20:31.227708 8554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:34367
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:34367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42935/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:31.228615 8554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:31.229528 8554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:20:31.237565 8554 server_base.cc:1034] running on GCE node
W20260810 06:20:31.237510 8560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:31.237500 8564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:31.237530 8566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:31.238332 8554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:31.239344 8554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:31.240517 8554 hybrid_clock.cc:648] HybridClock initialized: now 1786342831240496 us; error 29 us; skew 500 ppm
I20260810 06:20:31.242376 8554 webserver.cc:460] Webserver started at http://127.31.204.254:36965/ using document root <none> and password file <none>
I20260810 06:20:31.242892 8554 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:31.242954 8554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:31.243153 8554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:31.245071 8554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "5301f4d5a7894fd1ba851e91cf3c9a4c"
format_stamp: "Formatted at 2026-08-10 06:20:31 on dist-test-slave-q9h9"
server_key: "00611d6ccb78d46d81beca822d35ff48"
server_key_iv: "0d89f65a6cb24e12169c65a600db63f1"
server_key_version: "encryptionkey@0"
I20260810 06:20:31.245684 8554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "5301f4d5a7894fd1ba851e91cf3c9a4c"
format_stamp: "Formatted at 2026-08-10 06:20:31 on dist-test-slave-q9h9"
server_key: "00611d6ccb78d46d81beca822d35ff48"
server_key_iv: "0d89f65a6cb24e12169c65a600db63f1"
server_key_version: "encryptionkey@0"
I20260810 06:20:31.249222 8554 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:20:31.251462 8574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:31.252561 8554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:31.252674 8554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "5301f4d5a7894fd1ba851e91cf3c9a4c"
format_stamp: "Formatted at 2026-08-10 06:20:31 on dist-test-slave-q9h9"
server_key: "00611d6ccb78d46d81beca822d35ff48"
server_key_iv: "0d89f65a6cb24e12169c65a600db63f1"
server_key_version: "encryptionkey@0"
I20260810 06:20:31.252764 8554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:31.269155 8554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:31.269732 8554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:31.269896 8554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:31.276373 8554 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:34367
I20260810 06:20:31.276382 8666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:34367 every 8 connection(s)
I20260810 06:20:31.277215 8554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:20:31.277496 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 8554
I20260810 06:20:31.277601 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:20:31.277912 32563 external_mini_cluster.cc:1428] Setting key 2a4b3746e152fe47ab94e0a8071fd562
I20260810 06:20:31.280498 8667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:31.285540 8667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c: Bootstrap starting.
I20260810 06:20:31.286547 8671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:31.279228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32928 (local address 127.31.204.254:34367)
0810 06:20:31.279833 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:31.279848 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:31.279881 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:20:31.280492 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:31.280507 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:31.280909 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:31.281213 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:31.281613 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:31.282599 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:31.283475 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:31.283920 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:31.284067 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:31.284882 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:31.284887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:31.284898 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:20:31.285254 (+ 356us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:31.285306 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:20:31.285326 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:31.285411 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:31.285570 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 06:20:31.285831 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":125,"threads_started":1}
I20260810 06:20:31.287876 8667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:31.288679 8667 log.cc:824] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:31.289249 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:20:31.290582 8667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c: No bootstrap required, opened a new log
I20260810 06:20:31.293854 8667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5301f4d5a7894fd1ba851e91cf3c9a4c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34367 } }
I20260810 06:20:31.294091 8667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:31.294147 8667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5301f4d5a7894fd1ba851e91cf3c9a4c, State: Initialized, Role: FOLLOWER
I20260810 06:20:31.294486 8667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5301f4d5a7894fd1ba851e91cf3c9a4c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34367 } }
I20260810 06:20:31.294596 8667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:20:31.294641 8667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:20:31.294708 8667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:31.295604 8667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5301f4d5a7894fd1ba851e91cf3c9a4c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34367 } }
I20260810 06:20:31.295921 8667 leader_election.cc:302] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5301f4d5a7894fd1ba851e91cf3c9a4c; no voters:
I20260810 06:20:31.296200 8667 leader_election.cc:288] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:20:31.296312 8674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:31.296518 8674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c [term 1 LEADER]: Becoming Leader. State: Replica: 5301f4d5a7894fd1ba851e91cf3c9a4c, State: Running, Role: LEADER
I20260810 06:20:31.296788 8674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5301f4d5a7894fd1ba851e91cf3c9a4c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34367 } }
I20260810 06:20:31.297279 8667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:31.298444 8675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5301f4d5a7894fd1ba851e91cf3c9a4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5301f4d5a7894fd1ba851e91cf3c9a4c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34367 } } }
I20260810 06:20:31.298444 8677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5301f4d5a7894fd1ba851e91cf3c9a4c. Latest consensus state: current_term: 1 leader_uuid: "5301f4d5a7894fd1ba851e91cf3c9a4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5301f4d5a7894fd1ba851e91cf3c9a4c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34367 } } }
I20260810 06:20:31.298686 8675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:31.298713 8677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:31.299449 8707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:31.302459 8707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:31.307092 8707 catalog_manager.cc:1347] Generated new cluster ID: 4471be136b204021a8d016e0a5cbf234
I20260810 06:20:31.307152 8707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:31.322876 8707 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:20:31.323798 8707 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:31.331749 8707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5301f4d5a7894fd1ba851e91cf3c9a4c: Generated new TSK 0
I20260810 06:20:31.332356 8707 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 06:20:31.368863 8732 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 06:20:31.369004 8732 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 06:20:31.369408 8671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:31.351244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32934 (local address 127.31.204.254:34367)
0810 06:20:31.351376 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:31.351381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:31.351577 (+ 196us) server_negotiation.cc:407] Connection header received
0810 06:20:31.351785 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:31.351788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:31.351830 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:31.351906 (+ 76us) server_negotiation.cc:226] Negotiated authn=JWT
0810 06:20:31.352199 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:31.352702 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:31.353876 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:31.354119 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:31.354190 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:31.354528 (+ 338us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 06:20:31.369106 (+ 14578us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 06:20:31.369311 (+ 205us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32934: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 06:20:31.369455 8671 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32934: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 06:20:31.370958 8729 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:31.351128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34367 (local address 127.0.0.1:32934)
0810 06:20:31.351494 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:31.351509 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:31.351639 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:31.351941 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:31.351945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:31.351953 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 06:20:31.352098 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:31.352104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:31.352794 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:31.352798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:31.353774 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:31.353784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:31.354216 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:31.354220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:31.354314 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:31.354364 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 06:20:31.369317 (+ 14953us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 06:20:31.369533 (+ 216us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.204.254:34367: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":117,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.31.204.254:34367: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 06:20:31.371588 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 8554
2026-08-10T06:20:31Z chronyd exiting
I20260810 06:20:31.416483 32563 test_util.cc:175] -----------------------------------------------
I20260810 06:20:31.416548 32563 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786342803670820-32563-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (316 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 06:20:31.417634 32563 mini_oidc.cc:152] Starting JWKS server
I20260810 06:20:31.418501 32563 webserver.cc:460] Webserver started at https://127.0.0.1:44243/ using document root <none> and password file <none>
I20260810 06:20:31.418586 32563 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 06:20:31.418792 32563 webserver.cc:460] Webserver started at http://127.0.0.1:40935/ using document root <none> and password file <none>
2026-08-10T06:20:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:20:31Z Disabled control of system clock
I20260810 06:20:31.428035 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:34845
--webserver_interface=127.31.204.254
--webserver_port=0
--builtin_ntp_servers=127.31.204.212:37191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:34845
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44243/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786342803670820-32563-0/testchainca.cert with env {}
W20260810 06:20:31.535799 8746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:31.536010 8746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:31.536064 8746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:31.541332 8746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:31.541397 8746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:31.541419 8746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:31.541450 8746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:31.541479 8746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 06:20:31.541508 8746 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 06:20:31.541534 8746 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44243/jwks.json
I20260810 06:20:31.548676 8746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:37191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:34845
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:34845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44243/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786342803670820-32563-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:31.550840 8746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:31.552027 8746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:31.561597 8755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:31.561606 8756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:31.561606 8758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:31.562186 8746 server_base.cc:1034] running on GCE node
I20260810 06:20:31.562726 8746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:31.563769 8746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:31.564934 8746 hybrid_clock.cc:648] HybridClock initialized: now 1786342831564905 us; error 47 us; skew 500 ppm
I20260810 06:20:31.566654 8746 webserver.cc:460] Webserver started at http://127.31.204.254:38935/ using document root <none> and password file <none>
I20260810 06:20:31.567162 8746 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:31.567229 8746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:31.567400 8746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:31.569394 8746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "b067c0c197e34475b655257058f4b8ed"
format_stamp: "Formatted at 2026-08-10 06:20:31 on dist-test-slave-q9h9"
server_key: "d8d6515dbdcbc13aafb65a2cb60b45ad"
server_key_iv: "f41dda0d0784ab87f1cfbdf935be30ad"
server_key_version: "encryptionkey@0"
I20260810 06:20:31.570009 8746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "b067c0c197e34475b655257058f4b8ed"
format_stamp: "Formatted at 2026-08-10 06:20:31 on dist-test-slave-q9h9"
server_key: "d8d6515dbdcbc13aafb65a2cb60b45ad"
server_key_iv: "f41dda0d0784ab87f1cfbdf935be30ad"
server_key_version: "encryptionkey@0"
I20260810 06:20:31.574165 8746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 06:20:31.576313 8766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:31.577371 8746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:31.577478 8746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "b067c0c197e34475b655257058f4b8ed"
format_stamp: "Formatted at 2026-08-10 06:20:31 on dist-test-slave-q9h9"
server_key: "d8d6515dbdcbc13aafb65a2cb60b45ad"
server_key_iv: "f41dda0d0784ab87f1cfbdf935be30ad"
server_key_version: "encryptionkey@0"
I20260810 06:20:31.577571 8746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:31.589466 8746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:31.590000 8746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:31.590169 8746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:31.596424 8746 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:34845
I20260810 06:20:31.596465 8874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:34845 every 8 connection(s)
I20260810 06:20:31.597203 8746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:20:31.600106 8875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:31.603833 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 8746
I20260810 06:20:31.603950 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:20:31.604241 32563 external_mini_cluster.cc:1428] Setting key f2fc7b7797e1eb10859c70069c216f87
I20260810 06:20:31.605710 8875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed: Bootstrap starting.
I20260810 06:20:31.608201 8875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:31.609012 8875 log.cc:824] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:31.611078 8875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed: No bootstrap required, opened a new log
I20260810 06:20:31.613880 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:31.605768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.31.204.254:34845)
0810 06:20:31.606537 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:31.606564 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:31.606606 (+ 42us) server_negotiation.cc:407] Connection header received
0810 06:20:31.607332 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:31.607351 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:31.607799 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:31.608127 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:31.608613 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:31.609719 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:31.610713 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:31.611201 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:31.611395 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:31.612199 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:31.612205 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:31.612216 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:20:31.612595 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:31.612652 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:20:31.612672 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:31.612765 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:31.612916 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 06:20:31.613173 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":151,"threads_started":1}
I20260810 06:20:31.614632 8875 raft_consensus.cc:348] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b067c0c197e34475b655257058f4b8ed" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34845 } }
I20260810 06:20:31.614910 8875 raft_consensus.cc:374] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:31.614969 8875 raft_consensus.cc:729] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b067c0c197e34475b655257058f4b8ed, State: Initialized, Role: FOLLOWER
I20260810 06:20:31.615402 8875 consensus_queue.cc:260] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b067c0c197e34475b655257058f4b8ed" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34845 } }
I20260810 06:20:31.615549 8875 raft_consensus.cc:388] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:20:31.615612 8875 raft_consensus.cc:482] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:20:31.615706 8875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:31.616677 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:20:31.616811 8875 raft_consensus.cc:504] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b067c0c197e34475b655257058f4b8ed" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34845 } }
I20260810 06:20:31.617113 8875 leader_election.cc:302] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b067c0c197e34475b655257058f4b8ed; no voters:
I20260810 06:20:31.617395 8875 leader_election.cc:288] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:20:31.617506 8881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:31.617741 8881 raft_consensus.cc:686] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed [term 1 LEADER]: Becoming Leader. State: Replica: b067c0c197e34475b655257058f4b8ed, State: Running, Role: LEADER
I20260810 06:20:31.618074 8881 consensus_queue.cc:237] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b067c0c197e34475b655257058f4b8ed" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34845 } }
I20260810 06:20:31.618657 8875 sys_catalog.cc:564] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:31.619625 8883 sys_catalog.cc:455] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader b067c0c197e34475b655257058f4b8ed. Latest consensus state: current_term: 1 leader_uuid: "b067c0c197e34475b655257058f4b8ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b067c0c197e34475b655257058f4b8ed" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34845 } } }
I20260810 06:20:31.619625 8882 sys_catalog.cc:455] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b067c0c197e34475b655257058f4b8ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b067c0c197e34475b655257058f4b8ed" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34845 } } }
I20260810 06:20:31.619797 8883 sys_catalog.cc:458] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:31.619797 8882 sys_catalog.cc:458] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:31.620154 8895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:31.623026 8895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:31.629424 8895 catalog_manager.cc:1347] Generated new cluster ID: 4f5c05e0a15744feb9f874caa21aaf6d
I20260810 06:20:31.629479 8895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:31.668187 8895 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:20:31.669427 8895 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:31.677731 8895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b067c0c197e34475b655257058f4b8ed: Generated new TSK 0
I20260810 06:20:31.678411 8895 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 06:20:31.700383 8918 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 06:20:31.700459 8918 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 06:20:31.700996 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:31.686128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.31.204.254:34845)
0810 06:20:31.686254 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:31.686259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:31.686445 (+ 186us) server_negotiation.cc:407] Connection header received
0810 06:20:31.686586 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:31.686589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:31.686638 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:31.686726 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0810 06:20:31.686972 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:31.687490 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:31.688358 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:31.688577 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:31.688661 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:31.688975 (+ 314us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 06:20:31.700660 (+ 11685us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 06:20:31.700877 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47666: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 06:20:31.701058 8879 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47666: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 06:20:31.701104 8916 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:31.686076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34845 (local address 127.0.0.1:47666)
0810 06:20:31.686361 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:31.686372 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:31.686477 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:31.686744 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:31.686747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:31.686755 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 06:20:31.686869 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:31.686874 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:31.687619 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:31.687622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:31.688241 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:31.688249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:31.688686 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:31.688700 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:31.688787 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:31.688810 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 06:20:31.700810 (+ 12000us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 06:20:31.700954 (+ 144us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.204.254:34845: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":106,"threads_started":1}
I20260810 06:20:31.701795 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 8746
2026-08-10T06:20:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (336 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 06:20:31.754163 32563 mini_oidc.cc:152] Starting JWKS server
I20260810 06:20:31.754976 32563 webserver.cc:460] Webserver started at https://127.0.0.1:45039/ using document root <none> and password file <none>
I20260810 06:20:31.755054 32563 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 06:20:31.755250 32563 webserver.cc:460] Webserver started at http://127.0.0.1:39641/ using document root <none> and password file <none>
2026-08-10T06:20:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:20:31Z Disabled control of system clock
I20260810 06:20:31.764155 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:40657
--webserver_interface=127.31.204.254
--webserver_port=0
--builtin_ntp_servers=127.31.204.212:35777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:40657
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45039/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 06:20:31.870428 8927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:31.870635 8927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:31.870669 8927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:31.874441 8927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:31.874485 8927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:31.874498 8927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:31.874513 8927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:31.874529 8927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 06:20:31.874548 8927 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 06:20:31.874563 8927 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45039/jwks.json
I20260810 06:20:31.878989 8927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:40657
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:40657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45039/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:31.879803 8927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:31.880956 8927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:31.888545 8934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:31.888545 8933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:31.888541 8938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:31.888702 8927 server_base.cc:1034] running on GCE node
I20260810 06:20:31.889313 8927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:31.890314 8927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:31.891472 8927 hybrid_clock.cc:648] HybridClock initialized: now 1786342831891456 us; error 32 us; skew 500 ppm
I20260810 06:20:31.893096 8927 webserver.cc:460] Webserver started at http://127.31.204.254:46823/ using document root <none> and password file <none>
I20260810 06:20:31.893592 8927 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:31.893657 8927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:31.893854 8927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:31.895912 8927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "d6e9d7e4289e409991fb112d8371c144"
format_stamp: "Formatted at 2026-08-10 06:20:31 on dist-test-slave-q9h9"
server_key: "67324c130e0ec1ef7d253e5ad0f0490b"
server_key_iv: "49ce417fbcd681aa9dc44d0924a3013d"
server_key_version: "encryptionkey@0"
I20260810 06:20:31.896481 8927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "d6e9d7e4289e409991fb112d8371c144"
format_stamp: "Formatted at 2026-08-10 06:20:31 on dist-test-slave-q9h9"
server_key: "67324c130e0ec1ef7d253e5ad0f0490b"
server_key_iv: "49ce417fbcd681aa9dc44d0924a3013d"
server_key_version: "encryptionkey@0"
I20260810 06:20:31.899959 8927 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:20:31.902055 8950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:31.903136 8927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:31.903240 8927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "d6e9d7e4289e409991fb112d8371c144"
format_stamp: "Formatted at 2026-08-10 06:20:31 on dist-test-slave-q9h9"
server_key: "67324c130e0ec1ef7d253e5ad0f0490b"
server_key_iv: "49ce417fbcd681aa9dc44d0924a3013d"
server_key_version: "encryptionkey@0"
I20260810 06:20:31.903331 8927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:31.920572 8927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:31.921135 8927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:31.921296 8927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:31.927831 8927 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:40657
I20260810 06:20:31.927851 9037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:40657 every 8 connection(s)
I20260810 06:20:31.928637 8927 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:20:31.928926 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 8927
I20260810 06:20:31.929029 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:20:31.929334 32563 external_mini_cluster.cc:1428] Setting key 4d1866392424ebc5570f1470fada6321
I20260810 06:20:31.931614 9038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:31.936980 9038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144: Bootstrap starting.
I20260810 06:20:31.938313 9044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:31.930687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49366 (local address 127.31.204.254:40657)
0810 06:20:31.931311 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:31.931326 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:31.931359 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:20:31.931989 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:31.932005 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:31.932381 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:31.932672 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:31.933070 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:31.934176 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:31.935070 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:31.935518 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:31.935657 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:31.936502 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:31.936507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:31.936518 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:20:31.936863 (+ 345us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:31.936913 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:20:31.936932 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:31.937016 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:31.937192 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 06:20:31.937447 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":145,"threads_started":1}
I20260810 06:20:31.939510 9038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:31.940256 9038 log.cc:824] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:31.940932 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:20:31.942111 9038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144: No bootstrap required, opened a new log
I20260810 06:20:31.945334 9038 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e9d7e4289e409991fb112d8371c144" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 40657 } }
I20260810 06:20:31.945526 9038 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:31.945575 9038 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e9d7e4289e409991fb112d8371c144, State: Initialized, Role: FOLLOWER
I20260810 06:20:31.945948 9038 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e9d7e4289e409991fb112d8371c144" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 40657 } }
I20260810 06:20:31.946096 9038 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:20:31.946157 9038 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:20:31.946250 9038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:31.947113 9038 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e9d7e4289e409991fb112d8371c144" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 40657 } }
I20260810 06:20:31.947435 9038 leader_election.cc:302] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6e9d7e4289e409991fb112d8371c144; no voters:
I20260810 06:20:31.947705 9038 leader_election.cc:288] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:20:31.947778 9050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:31.947985 9050 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144 [term 1 LEADER]: Becoming Leader. State: Replica: d6e9d7e4289e409991fb112d8371c144, State: Running, Role: LEADER
I20260810 06:20:31.948294 9050 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e9d7e4289e409991fb112d8371c144" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 40657 } }
I20260810 06:20:31.948741 9038 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:31.949721 9052 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6e9d7e4289e409991fb112d8371c144" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e9d7e4289e409991fb112d8371c144" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 40657 } } }
I20260810 06:20:31.949855 9052 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144 [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:31.950039 9053 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6e9d7e4289e409991fb112d8371c144. Latest consensus state: current_term: 1 leader_uuid: "d6e9d7e4289e409991fb112d8371c144" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e9d7e4289e409991fb112d8371c144" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 40657 } } }
I20260810 06:20:31.950250 9053 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144 [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:31.950829 9063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:31.953239 9063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:31.958268 9063 catalog_manager.cc:1347] Generated new cluster ID: a9a20746c5294156a36a1969dcbfd830
I20260810 06:20:31.958328 9063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:31.983887 9063 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:20:31.984656 9063 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:31.997669 9063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6e9d7e4289e409991fb112d8371c144: Generated new TSK 0
I20260810 06:20:31.998256 9063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:20:32.009019 9044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:32.004740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49376 (local address 127.31.204.254:40657)
0810 06:20:32.004853 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:32.004857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:32.005047 (+ 190us) server_negotiation.cc:407] Connection header received
0810 06:20:32.005198 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:32.005201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:32.005245 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:32.005331 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0810 06:20:32.005565 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:32.006101 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:32.007042 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:32.007247 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:32.007332 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:32.007651 (+ 319us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 06:20:32.008690 (+ 1039us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45039/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 06:20:32.008902 (+ 212us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49376: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45039/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 06:20:32.009176 9044 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49376: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45039/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 06:20:32.009289 9084 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:32.004689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:40657 (local address 127.0.0.1:49376)
0810 06:20:32.004966 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:32.004977 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:32.005082 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:32.005350 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:32.005353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:32.005360 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0810 06:20:32.005464 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:32.005470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:32.006210 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:32.006213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:32.006929 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:32.006940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:32.007375 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:32.007379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:32.007458 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:32.007481 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 06:20:32.008839 (+ 1358us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45039/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 06:20:32.008971 (+ 132us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.204.254:40657: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45039/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":101,"threads_started":1}
I20260810 06:20:32.009949 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 8927
E20260810 06:20:32.026250 9085 webserver.cc:533] Webserver: sslize failed:
E20260810 06:20:32.026311 9085 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T06:20:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (300 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T06:20:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:20:32Z Disabled control of system clock
I20260810 06:20:32.063292 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:45077
--webserver_interface=127.31.204.254
--webserver_port=0
--builtin_ntp_servers=127.31.204.212:44791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:45077
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786342803670820-32563-0/testchainca.cert with env {}
W20260810 06:20:32.167371 9090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:32.167588 9090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:32.167623 9090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:32.171415 9090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:32.171458 9090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:32.171473 9090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:32.171486 9090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:32.171504 9090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 06:20:32.171520 9090 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 06:20:32.171535 9090 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 06:20:32.176465 9090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:44791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:45077
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:45077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786342803670820-32563-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:32.177282 9090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:32.178144 9090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:32.186266 9100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:32.186338 9090 server_base.cc:1034] running on GCE node
W20260810 06:20:32.186267 9099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:32.186302 9104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:32.186964 9090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:32.187965 9090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:32.189126 9090 hybrid_clock.cc:648] HybridClock initialized: now 1786342832189108 us; error 31 us; skew 500 ppm
I20260810 06:20:32.190809 9090 webserver.cc:460] Webserver started at http://127.31.204.254:45995/ using document root <none> and password file <none>
I20260810 06:20:32.191309 9090 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:32.191375 9090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:32.191550 9090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:32.193730 9090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "c1c99047776a46aeb4b74d837afc93c5"
format_stamp: "Formatted at 2026-08-10 06:20:32 on dist-test-slave-q9h9"
server_key: "eb29a7ad54fa637218dde7300a9fab2d"
server_key_iv: "969b2c97ef802b6dd8f7a07d5f978c7d"
server_key_version: "encryptionkey@0"
I20260810 06:20:32.194271 9090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "c1c99047776a46aeb4b74d837afc93c5"
format_stamp: "Formatted at 2026-08-10 06:20:32 on dist-test-slave-q9h9"
server_key: "eb29a7ad54fa637218dde7300a9fab2d"
server_key_iv: "969b2c97ef802b6dd8f7a07d5f978c7d"
server_key_version: "encryptionkey@0"
I20260810 06:20:32.197638 9090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260810 06:20:32.199853 9114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:32.200915 9090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:32.201032 9090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "c1c99047776a46aeb4b74d837afc93c5"
format_stamp: "Formatted at 2026-08-10 06:20:32 on dist-test-slave-q9h9"
server_key: "eb29a7ad54fa637218dde7300a9fab2d"
server_key_iv: "969b2c97ef802b6dd8f7a07d5f978c7d"
server_key_version: "encryptionkey@0"
I20260810 06:20:32.201120 9090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:32.222345 9090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:32.222877 9090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:32.223039 9090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:32.229384 9090 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:45077
I20260810 06:20:32.229411 9201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:45077 every 8 connection(s)
I20260810 06:20:32.230233 9090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:20:32.232684 9203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:32.237677 9203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5: Bootstrap starting.
I20260810 06:20:32.238150 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 9090
I20260810 06:20:32.238267 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:20:32.238523 32563 external_mini_cluster.cc:1428] Setting key c1038d877ed0495832f7cd1a20b58107
I20260810 06:20:32.240093 9203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:32.240777 9203 log.cc:824] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:32.242596 9203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5: No bootstrap required, opened a new log
I20260810 06:20:32.246430 9203 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c99047776a46aeb4b74d837afc93c5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45077 } }
I20260810 06:20:32.246637 9203 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:32.246701 9203 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c99047776a46aeb4b74d837afc93c5, State: Initialized, Role: FOLLOWER
I20260810 06:20:32.246975 9209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:32.239673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.31.204.254:45077)
0810 06:20:32.240255 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:32.240270 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:32.240303 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:20:32.240861 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:32.240877 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:32.241259 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:32.241516 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:32.241821 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:32.242827 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:32.244014 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:32.244467 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:32.244585 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:32.245338 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:32.245342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:32.245353 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:20:32.245683 (+ 330us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:32.245732 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:20:32.245749 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:32.245823 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:32.245957 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 06:20:32.246237 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":100,"threads_started":1}
I20260810 06:20:32.247134 9203 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c99047776a46aeb4b74d837afc93c5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45077 } }
I20260810 06:20:32.247267 9203 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:20:32.247334 9203 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:20:32.247423 9203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:32.248153 9203 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c99047776a46aeb4b74d837afc93c5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45077 } }
I20260810 06:20:32.248451 9203 leader_election.cc:302] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1c99047776a46aeb4b74d837afc93c5; no voters:
I20260810 06:20:32.248721 9203 leader_election.cc:288] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:20:32.248812 9212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:32.249046 9212 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5 [term 1 LEADER]: Becoming Leader. State: Replica: c1c99047776a46aeb4b74d837afc93c5, State: Running, Role: LEADER
I20260810 06:20:32.249380 9212 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c99047776a46aeb4b74d837afc93c5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45077 } }
I20260810 06:20:32.249642 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:20:32.249867 9203 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:32.251658 9213 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1c99047776a46aeb4b74d837afc93c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c99047776a46aeb4b74d837afc93c5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45077 } } }
I20260810 06:20:32.251793 9213 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5 [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:32.252099 9216 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1c99047776a46aeb4b74d837afc93c5. Latest consensus state: current_term: 1 leader_uuid: "c1c99047776a46aeb4b74d837afc93c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c99047776a46aeb4b74d837afc93c5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45077 } } }
I20260810 06:20:32.252345 9216 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5 [sys.catalog]: This master's current role is: LEADER
W20260810 06:20:32.254359 9234 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:20:32.254420 9234 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:20:32.254506 9236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:32.255403 9236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:32.260862 9236 catalog_manager.cc:1347] Generated new cluster ID: d767110aa6c84bcd8df60fcb59f37d45
I20260810 06:20:32.260916 9236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:32.279783 9236 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:20:32.280869 9236 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:32.288535 9236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1c99047776a46aeb4b74d837afc93c5: Generated new TSK 0
I20260810 06:20:32.289081 9236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:20:32.348169 9209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:32.296560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43468 (local address 127.31.204.254:45077)
0810 06:20:32.296677 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:32.296682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:32.296835 (+ 153us) server_negotiation.cc:407] Connection header received
0810 06:20:32.296950 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:32.296953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:32.297005 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:32.297110 (+ 105us) server_negotiation.cc:226] Negotiated authn=JWT
0810 06:20:32.297328 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:32.298150 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:32.298966 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:32.299311 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:32.299400 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:32.299643 (+ 243us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 06:20:32.347741 (+ 48098us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 06:20:32.348024 (+ 283us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43468: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 06:20:32.348244 9209 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43468: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 06:20:32.348275 9250 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:32.296440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:45077 (local address 127.0.0.1:43468)
0810 06:20:32.296753 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:32.296765 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:32.296858 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:32.297110 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:32.297112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:32.297118 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0810 06:20:32.297230 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:32.297235 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:32.298253 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:32.298256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:32.298863 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:32.298870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:32.299419 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:32.299422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:32.299485 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:32.299505 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 06:20:32.347968 (+ 48463us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 06:20:32.348122 (+ 154us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.204.254:45077: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":116,"threads_started":1}
I20260810 06:20:32.349068 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 9090
2026-08-10T06:20:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (320 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (119 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (115 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:20:32 dist-test-slave-q9h9 krb5kdc[9266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:20:32 dist-test-slave-q9h9 krb5kdc[9266](info): set up 2 sockets
Aug 10 06:20:32 dist-test-slave-q9h9 krb5kdc[9266](info): commencing operation
krb5kdc: starting...
W20260810 06:20:33.746292 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:20:33 dist-test-slave-q9h9 krb5kdc[9266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:20:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:20:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:20:33.858913 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:35811
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:44595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:35811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:33.963107 9289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:33.963331 9289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:33.963367 9289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:33.967154 9289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:33.967196 9289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:33.967213 9289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:33.967233 9289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:33.967249 9289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:20:33.972117 9289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:44595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:35811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:35811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:33.972965 9289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:33.973850 9289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:33.981702 9296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:33.981778 9309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:33.981769 9289 server_base.cc:1034] running on GCE node
W20260810 06:20:33.981695 9300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:33.982430 9289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:33.983457 9289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:33.984638 9289 hybrid_clock.cc:648] HybridClock initialized: now 1786342833984614 us; error 33 us; skew 500 ppm
Aug 10 06:20:33 dist-test-slave-q9h9 krb5kdc[9266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342833, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:33.987313 9289 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:20:33.988139 9289 webserver.cc:460] Webserver started at http://127.31.204.254:33379/ using document root <none> and password file <none>
I20260810 06:20:33.988651 9289 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:33.988714 9289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:33.988898 9289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:33.990911 9289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "b22b3c9d53c04cd8a3e7756bbb4c261f"
format_stamp: "Formatted at 2026-08-10 06:20:33 on dist-test-slave-q9h9"
server_key: "fa298a87131391a3f6f2dfbbaba83ac5"
server_key_iv: "c15e830d3912b41c2aeca4e04910fa81"
server_key_version: "encryptionkey@0"
I20260810 06:20:33.991453 9289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "b22b3c9d53c04cd8a3e7756bbb4c261f"
format_stamp: "Formatted at 2026-08-10 06:20:33 on dist-test-slave-q9h9"
server_key: "fa298a87131391a3f6f2dfbbaba83ac5"
server_key_iv: "c15e830d3912b41c2aeca4e04910fa81"
server_key_version: "encryptionkey@0"
I20260810 06:20:33.994936 9289 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:20:33.997172 9317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:33.998328 9289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:33.998445 9289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "b22b3c9d53c04cd8a3e7756bbb4c261f"
format_stamp: "Formatted at 2026-08-10 06:20:33 on dist-test-slave-q9h9"
server_key: "fa298a87131391a3f6f2dfbbaba83ac5"
server_key_iv: "c15e830d3912b41c2aeca4e04910fa81"
server_key_version: "encryptionkey@0"
I20260810 06:20:33.998539 9289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:34.017756 9289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:34.020386 9289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:34.020553 9289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:34.027338 9289 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:35811
I20260810 06:20:34.027356 9410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:35811 every 8 connection(s)
I20260810 06:20:34.028216 9289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:20:34.030809 9411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:34.033826 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 9289
I20260810 06:20:34.033962 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:20:34.034284 32563 external_mini_cluster.cc:1428] Setting key d003a0ad3939bb89dcd8f591818210ef
I20260810 06:20:34.036479 9411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f: Bootstrap starting.
I20260810 06:20:34.039041 9411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:34.039861 9411 log.cc:824] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:34.041867 9411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f: No bootstrap required, opened a new log
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[9266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:34.045737 9411 raft_consensus.cc:348] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22b3c9d53c04cd8a3e7756bbb4c261f" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35811 } }
I20260810 06:20:34.045986 9411 raft_consensus.cc:374] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:34.046063 9411 raft_consensus.cc:729] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22b3c9d53c04cd8a3e7756bbb4c261f, State: Initialized, Role: FOLLOWER
I20260810 06:20:34.046499 9411 consensus_queue.cc:260] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22b3c9d53c04cd8a3e7756bbb4c261f" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35811 } }
I20260810 06:20:34.046654 9411 raft_consensus.cc:388] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:20:34.046723 9411 raft_consensus.cc:482] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:20:34.046815 9411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:34.047820 9411 raft_consensus.cc:504] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22b3c9d53c04cd8a3e7756bbb4c261f" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35811 } }
I20260810 06:20:34.048164 9411 leader_election.cc:302] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b22b3c9d53c04cd8a3e7756bbb4c261f; no voters:
I20260810 06:20:34.048481 9411 leader_election.cc:288] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:20:34.048565 9416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:34.048765 9416 raft_consensus.cc:686] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f [term 1 LEADER]: Becoming Leader. State: Replica: b22b3c9d53c04cd8a3e7756bbb4c261f, State: Running, Role: LEADER
I20260810 06:20:34.048925 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:34.035681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48878 (local address 127.31.204.254:35811)
0810 06:20:34.036086 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:34.036105 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:34.036147 (+ 42us) server_negotiation.cc:407] Connection header received
0810 06:20:34.036935 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:34.036958 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:34.037334 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:34.037687 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:34.038403 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.039833 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.040830 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.041468 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.041632 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:34.044101 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:34.044107 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:34.044121 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:34.044152 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:34.045953 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:34.046431 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:34.046436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:34.046442 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:34.046582 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:34.046858 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:34.046861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:34.046863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:34.047326 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:34.047519 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:34.047730 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 06:20:34.048011 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":107,"threads_started":1}
I20260810 06:20:34.049060 9416 consensus_queue.cc:237] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22b3c9d53c04cd8a3e7756bbb4c261f" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35811 } }
I20260810 06:20:34.049569 9411 sys_catalog.cc:564] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:34.050680 9418 sys_catalog.cc:455] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b22b3c9d53c04cd8a3e7756bbb4c261f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22b3c9d53c04cd8a3e7756bbb4c261f" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35811 } } }
I20260810 06:20:34.050796 9418 sys_catalog.cc:458] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:34.051055 9419 sys_catalog.cc:455] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b22b3c9d53c04cd8a3e7756bbb4c261f. Latest consensus state: current_term: 1 leader_uuid: "b22b3c9d53c04cd8a3e7756bbb4c261f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22b3c9d53c04cd8a3e7756bbb4c261f" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35811 } } }
I20260810 06:20:34.051419 9419 sys_catalog.cc:458] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f [sys.catalog]: This master's current role is: LEADER
W20260810 06:20:34.053938 9438 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:20:34.054009 9438 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:20:34.054108 9431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:34.054870 9431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:34.060047 9431 catalog_manager.cc:1347] Generated new cluster ID: 624cf6a7deb647f3add4d8c15a1f7c79
I20260810 06:20:34.060094 9431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:34.070315 9431 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:20:34.071089 9431 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:34.077153 9431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b22b3c9d53c04cd8a3e7756bbb4c261f: Generated new TSK 0
I20260810 06:20:34.077694 9431 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
I20260810 06:20:34.106344 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35811
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:44595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:34.214577 9448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:34.214833 9448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:34.214880 9448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:34.218873 9448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:34.218922 9448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:34.218979 9448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:20:34.223927 9448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:44595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:34.224864 9448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:34.225841 9448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:34.234952 9459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:34.234958 9458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:34.235109 9448 server_base.cc:1034] running on GCE node
W20260810 06:20:34.235267 9461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:34.235654 9448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:34.236255 9448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:34.237461 9448 hybrid_clock.cc:648] HybridClock initialized: now 1786342834237440 us; error 27 us; skew 500 ppm
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[9266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342834, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:34.240358 9448 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:20:34.241170 9448 webserver.cc:460] Webserver started at http://127.31.204.193:37307/ using document root <none> and password file <none>
I20260810 06:20:34.241662 9448 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:34.241721 9448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:34.241914 9448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:34.243835 9448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "95be34329e2f42dd84ab7df8c9ed160f"
format_stamp: "Formatted at 2026-08-10 06:20:34 on dist-test-slave-q9h9"
server_key: "d32521b3d83c4d67bc155a1f48d6397b"
server_key_iv: "29e678080ebcd3c53763976f6a4cb25a"
server_key_version: "encryptionkey@0"
I20260810 06:20:34.244359 9448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "95be34329e2f42dd84ab7df8c9ed160f"
format_stamp: "Formatted at 2026-08-10 06:20:34 on dist-test-slave-q9h9"
server_key: "d32521b3d83c4d67bc155a1f48d6397b"
server_key_iv: "29e678080ebcd3c53763976f6a4cb25a"
server_key_version: "encryptionkey@0"
I20260810 06:20:34.247891 9448 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:20:34.250221 9473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:34.251397 9448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:34.251487 9448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "95be34329e2f42dd84ab7df8c9ed160f"
format_stamp: "Formatted at 2026-08-10 06:20:34 on dist-test-slave-q9h9"
server_key: "d32521b3d83c4d67bc155a1f48d6397b"
server_key_iv: "29e678080ebcd3c53763976f6a4cb25a"
server_key_version: "encryptionkey@0"
I20260810 06:20:34.251561 9448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:34.276324 9448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:34.279212 9448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:34.279353 9448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:34.279830 9448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:34.280561 9448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:34.280606 9448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:34.280642 9448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:34.280661 9448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:34.288947 9448 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:33385
I20260810 06:20:34.288998 9677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:33385 every 8 connection(s)
I20260810 06:20:34.289959 9448 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:20:34.291831 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 9448
I20260810 06:20:34.291951 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:20:34.292239 32563 external_mini_cluster.cc:1428] Setting key f90f0b99f216674d963f703562fc1351
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[9266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342834, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.204.194@KRBTEST.COM; defaulting to no policy
I20260810 06:20:34.303925 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:34.291651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:54131 (local address 127.31.204.254:35811)
0810 06:20:34.291812 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:34.291818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:34.292609 (+ 791us) server_negotiation.cc:407] Connection header received
0810 06:20:34.294142 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:34.294145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:34.294197 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:34.294278 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:34.295371 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.296005 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.297492 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.297809 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.297911 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:34.300421 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:34.300425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:34.300428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:34.300459 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:34.301964 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:34.302520 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:34.302523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:34.302525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:34.302622 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:34.302992 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:34.302995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:34.302996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:34.303163 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:34.303238 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:34.303706 (+ 468us) server_negotiation.cc:299] Negotiation successful
0810 06:20:34.303807 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 06:20:34.304724 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:34.291996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:35811 (local address 127.31.204.193:54131)
0810 06:20:34.292467 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:34.292518 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:34.293916 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:34.294414 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:34.294420 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:34.294771 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:34.295223 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:34.295236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.296170 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.296174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:34.297356 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:34.297367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.297942 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.297944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:34.298143 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:34.298662 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:34.298677 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:34.300251 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:34.302097 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:34.302102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:34.302115 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:34.302419 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:34.302736 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:34.302739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:34.302740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:34.302875 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:34.303246 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:34.303250 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:34.303461 (+ 211us) client_negotiation.cc:769] Sending connection context
0810 06:20:34.303652 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 06:20:34.303892 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.194@KRBTEST.COM" created.
I20260810 06:20:34.305919 9678 heartbeater.cc:347] Connected to a master server at 127.31.204.254:35811
I20260810 06:20:34.306174 9678 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:34.306627 9678 heartbeater.cc:511] Master 127.31.204.254:35811 requested a full tablet report, sending...
I20260810 06:20:34.308149 9345 ts_manager.cc:194] Registered new tserver with Master: 95be34329e2f42dd84ab7df8c9ed160f (127.31.204.193:33385)
I20260810 06:20:34.309652 9345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:54131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
I20260810 06:20:34.319386 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35811
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:44595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:34.422842 9686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:34.423044 9686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:34.423082 9686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:34.426885 9686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:34.426927 9686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:34.426980 9686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:20:34.431766 9686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:44595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:34.432555 9686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:34.433377 9686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:34.441985 9696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:34.442050 9693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:34.442168 9686 server_base.cc:1034] running on GCE node
W20260810 06:20:34.441994 9701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:34.442660 9686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:34.443199 9686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:34.444358 9686 hybrid_clock.cc:648] HybridClock initialized: now 1786342834444331 us; error 44 us; skew 500 ppm
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[9266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342834, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:34.446887 9686 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:20:34.447697 9686 webserver.cc:460] Webserver started at http://127.31.204.194:41611/ using document root <none> and password file <none>
I20260810 06:20:34.448173 9686 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:34.448230 9686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:34.448436 9686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:34.450333 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "01c445d8cc5e4cbcb514e74c545e5292"
format_stamp: "Formatted at 2026-08-10 06:20:34 on dist-test-slave-q9h9"
server_key: "69dbbb729240733f8c406926a5b646d3"
server_key_iv: "ba8c9d06b29997476c2ef16ae6194965"
server_key_version: "encryptionkey@0"
I20260810 06:20:34.450891 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "01c445d8cc5e4cbcb514e74c545e5292"
format_stamp: "Formatted at 2026-08-10 06:20:34 on dist-test-slave-q9h9"
server_key: "69dbbb729240733f8c406926a5b646d3"
server_key_iv: "ba8c9d06b29997476c2ef16ae6194965"
server_key_version: "encryptionkey@0"
I20260810 06:20:34.454293 9686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:20:34.456534 9712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:34.457617 9686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:34.457706 9686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "01c445d8cc5e4cbcb514e74c545e5292"
format_stamp: "Formatted at 2026-08-10 06:20:34 on dist-test-slave-q9h9"
server_key: "69dbbb729240733f8c406926a5b646d3"
server_key_iv: "ba8c9d06b29997476c2ef16ae6194965"
server_key_version: "encryptionkey@0"
I20260810 06:20:34.457805 9686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:34.468291 9686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:34.470715 9686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:34.470875 9686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:34.471356 9686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:34.472065 9686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:34.472108 9686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:34.472144 9686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:34.472173 9686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:34.480398 9686 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:46711
I20260810 06:20:34.480439 9906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:46711 every 8 connection(s)
I20260810 06:20:34.481137 9686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:20:34.484781 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 9686
I20260810 06:20:34.484915 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:20:34.485194 32563 external_mini_cluster.cc:1428] Setting key 43f19158b86a5915a66a430c8f9c6cf9
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[9266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342834, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:34.493767 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:34.482689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:37417 (local address 127.31.204.254:35811)
0810 06:20:34.482852 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:34.482857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:34.483568 (+ 711us) server_negotiation.cc:407] Connection header received
0810 06:20:34.484794 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:34.484797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:34.484845 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:34.484928 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:34.486276 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.486959 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.488042 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.488341 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.488440 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:34.490596 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:34.490599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:34.490601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:34.490625 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:34.491873 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:34.492339 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:34.492342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:34.492343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:34.492420 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:34.492732 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:34.492736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:34.492738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:34.492916 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:34.493001 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:34.493483 (+ 482us) server_negotiation.cc:299] Negotiation successful
0810 06:20:34.493607 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 06:20:34.494477 9913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:34.483010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:35811 (local address 127.31.204.194:37417)
0810 06:20:34.483422 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:34.483460 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:34.484581 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:34.485177 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:34.485183 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:34.485531 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:34.486112 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:34.486131 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.487082 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.487085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:34.487915 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:34.487924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.488466 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.488468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:34.488623 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:34.489097 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:34.489112 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:34.490439 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:34.491981 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:34.491986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:34.491999 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:34.492233 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:34.492509 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:34.492512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:34.492515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:34.492638 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:34.493021 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:34.493027 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:34.493237 (+ 210us) client_negotiation.cc:769] Sending connection context
0810 06:20:34.493398 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 06:20:34.493682 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.31.204.195@KRBTEST.COM; defaulting to no policy
I20260810 06:20:34.495561 9909 heartbeater.cc:347] Connected to a master server at 127.31.204.254:35811
I20260810 06:20:34.495756 9909 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:34.496179 9909 heartbeater.cc:511] Master 127.31.204.254:35811 requested a full tablet report, sending...
I20260810 06:20:34.497064 9345 ts_manager.cc:194] Registered new tserver with Master: 01c445d8cc5e4cbcb514e74c545e5292 (127.31.204.194:46711)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.195@KRBTEST.COM" created.
I20260810 06:20:34.497632 9345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:37417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
I20260810 06:20:34.510896 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35811
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:44595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:34.616966 9918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:34.617195 9918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:34.617237 9918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:34.621155 9918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:34.621202 9918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:34.621255 9918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:20:34.626243 9918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:44595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:34.627115 9918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:34.628013 9918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:34.636837 9926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:34.636886 9918 server_base.cc:1034] running on GCE node
W20260810 06:20:34.636845 9924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:34.636842 9930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:34.637485 9918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:34.638088 9918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:34.639271 9918 hybrid_clock.cc:648] HybridClock initialized: now 1786342834639247 us; error 33 us; skew 500 ppm
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[9266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342834, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:34.641825 9918 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:20:34.642643 9918 webserver.cc:460] Webserver started at http://127.31.204.195:38895/ using document root <none> and password file <none>
I20260810 06:20:34.643131 9918 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:34.643194 9918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:34.643370 9918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:34.645200 9918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "22e289e01f5c43cda3474571b0237a86"
format_stamp: "Formatted at 2026-08-10 06:20:34 on dist-test-slave-q9h9"
server_key: "a4279fffc54779da796cbf2745e6f990"
server_key_iv: "222ce25143bc1dc1414af93db12277a0"
server_key_version: "encryptionkey@0"
I20260810 06:20:34.645720 9918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "22e289e01f5c43cda3474571b0237a86"
format_stamp: "Formatted at 2026-08-10 06:20:34 on dist-test-slave-q9h9"
server_key: "a4279fffc54779da796cbf2745e6f990"
server_key_iv: "222ce25143bc1dc1414af93db12277a0"
server_key_version: "encryptionkey@0"
I20260810 06:20:34.649159 9918 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 06:20:34.651477 9946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:34.652534 9918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:34.652661 9918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "22e289e01f5c43cda3474571b0237a86"
format_stamp: "Formatted at 2026-08-10 06:20:34 on dist-test-slave-q9h9"
server_key: "a4279fffc54779da796cbf2745e6f990"
server_key_iv: "222ce25143bc1dc1414af93db12277a0"
server_key_version: "encryptionkey@0"
I20260810 06:20:34.652751 9918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:34.668251 9918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:34.671085 9918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:34.671250 9918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:34.671761 9918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:34.672544 9918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:34.672590 9918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:34.672649 9918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:34.672680 9918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:34.680828 9918 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:36613
I20260810 06:20:34.680857 10140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:36613 every 8 connection(s)
I20260810 06:20:34.681571 9918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:20:34.686779 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 9918
I20260810 06:20:34.686877 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:20:34.687151 32563 external_mini_cluster.cc:1428] Setting key 8e0db5d5ef6d53f05346950d6fccd3ba
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[9266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342834, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:34.693773 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:34.683241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:51665 (local address 127.31.204.254:35811)
0810 06:20:34.683392 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:34.683395 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:34.684483 (+ 1088us) server_negotiation.cc:407] Connection header received
0810 06:20:34.685310 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:34.685313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:34.685371 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:34.685447 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:34.686599 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.687198 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.688489 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.688806 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.688907 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:34.691121 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:34.691124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:34.691126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:34.691149 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:34.692230 (+ 1081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:34.692663 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:34.692665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:34.692666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:34.692733 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:34.692983 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:34.692985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:34.692986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:34.693136 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:34.693203 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:34.693573 (+ 370us) server_negotiation.cc:299] Negotiation successful
0810 06:20:34.693666 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 06:20:34.694509 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:34.683542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:35811 (local address 127.31.204.195:51665)
0810 06:20:34.684344 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:34.684382 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:34.685113 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:34.685623 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:34.685630 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:34.685937 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:34.686426 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:34.686440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.687330 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.687333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:34.688356 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:34.688368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.688952 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.688954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:34.689142 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:34.689618 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:34.689634 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:34.690977 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:34.692333 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:34.692338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:34.692348 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:34.692586 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:34.692797 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:34.692799 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:34.692800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:34.692919 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:34.693208 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:34.693212 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:34.693402 (+ 190us) client_negotiation.cc:769] Sending connection context
0810 06:20:34.693542 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 06:20:34.693759 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":508,"threads_started":1}
I20260810 06:20:34.695492 10142 heartbeater.cc:347] Connected to a master server at 127.31.204.254:35811
I20260810 06:20:34.695688 10142 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:34.696087 10142 heartbeater.cc:511] Master 127.31.204.254:35811 requested a full tablet report, sending...
I20260810 06:20:34.696962 9345 ts_manager.cc:194] Registered new tserver with Master: 22e289e01f5c43cda3474571b0237a86 (127.31.204.195:36613)
I20260810 06:20:34.697474 9345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:51665
I20260810 06:20:34.700567 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/test-admin.keytab.
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[9266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:34.715498 32563 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 06:20:34.717957 10159 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:34.716618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52566 (local address 127.31.204.193:33385)
0810 06:20:34.716980 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:34.716988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:34.717009 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:20:34.717052 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:34.717056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:34.717214 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:34.717318 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:34.717787 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52566: BlockingRecv error: recv got EOF from 127.0.0.1:52566 (error 108)
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":157,"threads_started":1}
W20260810 06:20:34.718147 32563 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/krb5cc' of type FILE
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[9266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:34.719033 32563 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[9266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:34.719851 32563 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[9266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.193@KRBTEST.COM
I20260810 06:20:34.733505 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:34.720620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52576 (local address 127.31.204.193:33385)
0810 06:20:34.720730 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:34.720735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:34.720802 (+ 67us) server_negotiation.cc:407] Connection header received
0810 06:20:34.720915 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:34.720918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:34.720958 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:34.721019 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:34.721538 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.722321 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.723145 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.723474 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.723565 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:34.730387 (+ 6822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:34.730396 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:34.730406 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:34.730429 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:34.731832 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:34.732253 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:34.732258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:34.732259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:34.732337 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:34.732602 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:34.732605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:34.732607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:34.732823 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:34.732991 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:34.733237 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 06:20:34.733375 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 06:20:34.736811 10159 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:34.735945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52580 (local address 127.31.204.193:33385)
0810 06:20:34.736060 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:34.736064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:34.736115 (+ 51us) server_negotiation.cc:407] Connection header received
0810 06:20:34.736238 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:34.736241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:34.736278 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:34.736340 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:34.736723 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52580: BlockingRecv error: recv got EOF from 127.0.0.1:52580 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 06:20:34.736912 32563 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/krb5cc' of type FILE
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[9266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:34.737756 32563 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[9266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:34.738668 32563 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[9266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.193@KRBTEST.COM
I20260810 06:20:34.745601 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:34.739359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.31.204.193:33385)
0810 06:20:34.739481 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:34.739484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:34.739512 (+ 28us) server_negotiation.cc:407] Connection header received
0810 06:20:34.739604 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:34.739606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:34.739643 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:34.739701 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:34.740081 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.740531 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.741183 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.741382 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.741445 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:34.743281 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:34.743283 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:34.743285 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:34.743306 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:34.744480 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:34.744782 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:34.744785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:34.744786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:34.744849 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:34.745026 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:34.745028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:34.745029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:34.745159 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:34.745241 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:34.745388 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 06:20:34.745467 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 06:20:34.747766 10159 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:34.746968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52590 (local address 127.31.204.193:33385)
0810 06:20:34.747068 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:34.747072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:34.747137 (+ 65us) server_negotiation.cc:407] Connection header received
0810 06:20:34.747240 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:34.747243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:34.747288 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:34.747346 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:34.747690 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52590: BlockingRecv error: recv got EOF from 127.0.0.1:52590 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 06:20:34.748076 32563 init.cc:244] Error while opening credential cache '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786342803670820-32563-0/krb5kdc/krb5cc)
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[9266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:34.748930 32563 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[9266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:34.749740 32563 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[9266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.193@KRBTEST.COM
I20260810 06:20:34.757053 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:34.750492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52596 (local address 127.31.204.193:33385)
0810 06:20:34.750603 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:34.750607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:34.750691 (+ 84us) server_negotiation.cc:407] Connection header received
0810 06:20:34.750761 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:34.750764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:34.750793 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:34.750850 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:34.751285 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.751720 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.752394 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:34.752581 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:34.752641 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:34.754572 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:34.754576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:34.754578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:34.754598 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:34.755621 (+ 1023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:34.755999 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:34.756003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:34.756005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:34.756108 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:34.756360 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:34.756363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:34.756365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:34.756507 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:34.756622 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:34.756839 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 06:20:34.756929 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 06:20:34.757897 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 9448
I20260810 06:20:34.776230 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 9686
I20260810 06:20:34.789579 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 9918
I20260810 06:20:34.802145 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 9289
2026-08-10T06:20:34Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2213 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[10178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[10178](info): set up 2 sockets
Aug 10 06:20:34 dist-test-slave-q9h9 krb5kdc[10178](info): commencing operation
krb5kdc: starting...
W20260810 06:20:35.950922 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:20:36 dist-test-slave-q9h9 krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:20:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:20:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.254.keytab.
Entry for principal oryx/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.254.keytab.
I20260810 06:20:36.064962 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:34003
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.254.keytab
--principal=oryx/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:34003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:36.170830 10195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:36.171054 10195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:36.171118 10195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:36.175024 10195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:36.175081 10195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:36.175109 10195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:36.175132 10195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:36.175160 10195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:20:36.180090 10195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:34003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.254.keytab
--principal=oryx/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:34003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:36.180951 10195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:36.181854 10195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:36.189561 10201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:36.189592 10200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:36.189757 10195 server_base.cc:1034] running on GCE node
W20260810 06:20:36.189592 10203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:36.190438 10195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:36.191447 10195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:36.192617 10195 hybrid_clock.cc:648] HybridClock initialized: now 1786342836192599 us; error 32 us; skew 500 ppm
Aug 10 06:20:36 dist-test-slave-q9h9 krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342836, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:36.195312 10195 init.cc:377] Logged in from keytab as oryx/127.31.204.254@KRBTEST.COM (short username oryx)
I20260810 06:20:36.196161 10195 webserver.cc:460] Webserver started at http://127.31.204.254:32931/ using document root <none> and password file <none>
I20260810 06:20:36.196661 10195 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:36.196733 10195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:36.196920 10195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:36.198868 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "cf988718c8da472c975d2fa2b1e2a341"
format_stamp: "Formatted at 2026-08-10 06:20:36 on dist-test-slave-q9h9"
server_key: "c30e41fbb2d5d14cab07df5675a5f947"
server_key_iv: "726b1736b0a802a31bb342b155e7a11a"
server_key_version: "encryptionkey@0"
I20260810 06:20:36.199412 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "cf988718c8da472c975d2fa2b1e2a341"
format_stamp: "Formatted at 2026-08-10 06:20:36 on dist-test-slave-q9h9"
server_key: "c30e41fbb2d5d14cab07df5675a5f947"
server_key_iv: "726b1736b0a802a31bb342b155e7a11a"
server_key_version: "encryptionkey@0"
I20260810 06:20:36.202975 10195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:20:36.205142 10216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:36.206282 10195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:36.206393 10195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "cf988718c8da472c975d2fa2b1e2a341"
format_stamp: "Formatted at 2026-08-10 06:20:36 on dist-test-slave-q9h9"
server_key: "c30e41fbb2d5d14cab07df5675a5f947"
server_key_iv: "726b1736b0a802a31bb342b155e7a11a"
server_key_version: "encryptionkey@0"
I20260810 06:20:36.206481 10195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:36.235555 10195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:36.238385 10195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:36.238549 10195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:36.245164 10195 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:34003
I20260810 06:20:36.245400 10293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:34003 every 8 connection(s)
I20260810 06:20:36.246047 10195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:20:36.248557 10295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:36.250000 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 10195
I20260810 06:20:36.250161 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:20:36.250430 32563 external_mini_cluster.cc:1428] Setting key e9246bd198fffb66812df57c5f8fd36d
I20260810 06:20:36.254390 10295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341: Bootstrap starting.
I20260810 06:20:36.256707 10295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:36.257524 10295 log.cc:824] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341: Log is configured to *not* fsync() on all Append() calls
Aug 10 06:20:36 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM
I20260810 06:20:36.259433 10295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341: No bootstrap required, opened a new log
I20260810 06:20:36.262945 10295 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } }
I20260810 06:20:36.263134 10295 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:36.263164 10295 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf988718c8da472c975d2fa2b1e2a341, State: Initialized, Role: FOLLOWER
I20260810 06:20:36.263569 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:36.251611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40592 (local address 127.31.204.254:34003)
0810 06:20:36.252006 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:36.252021 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:36.252054 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:20:36.252679 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:36.252695 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:36.252979 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:36.253226 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:36.254131 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:36.255110 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:36.256273 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:36.256721 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:36.256872 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:36.259381 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:36.259386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:36.259397 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:36.259422 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:36.260927 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:36.261399 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:36.261403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:36.261407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:36.261504 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:36.261766 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:36.261768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:36.261769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:36.262153 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:36.262301 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:36.262573 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 06:20:36.262831 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":116,"threads_started":1}
I20260810 06:20:36.263535 10295 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } }
I20260810 06:20:36.263666 10295 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:20:36.263720 10295 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:20:36.263835 10295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:36.264665 10295 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } }
I20260810 06:20:36.264953 10295 leader_election.cc:302] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf988718c8da472c975d2fa2b1e2a341; no voters:
I20260810 06:20:36.265202 10295 leader_election.cc:288] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:20:36.265327 10306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:36.265509 10306 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [term 1 LEADER]: Becoming Leader. State: Replica: cf988718c8da472c975d2fa2b1e2a341, State: Running, Role: LEADER
I20260810 06:20:36.265789 10306 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } }
I20260810 06:20:36.266410 10295 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:36.267446 10308 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf988718c8da472c975d2fa2b1e2a341. Latest consensus state: current_term: 1 leader_uuid: "cf988718c8da472c975d2fa2b1e2a341" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } }
I20260810 06:20:36.267431 10307 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf988718c8da472c975d2fa2b1e2a341" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } }
I20260810 06:20:36.267567 10308 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:36.267578 10307 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:36.267906 10320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:36.270932 10320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:36.275861 10320 catalog_manager.cc:1347] Generated new cluster ID: a7dcab4da1e8485897cdfde29453d3a9
I20260810 06:20:36.275921 10320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:36.299149 10320 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:20:36.300001 10320 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:36.306211 10320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341: Generated new TSK 0
I20260810 06:20:36.306756 10320 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.193.keytab.
Entry for principal oryx/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.193.keytab.
I20260810 06:20:36.338294 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34003
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.193.keytab
--principal=oryx/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:36.442709 10346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:36.442921 10346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:36.442977 10346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:36.446810 10346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:36.446856 10346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:36.446930 10346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:20:36.451721 10346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.193.keytab
--principal=oryx/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:36.452567 10346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:36.453455 10346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:36.461979 10351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:36.462055 10357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:36.462136 10346 server_base.cc:1034] running on GCE node
W20260810 06:20:36.462157 10354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:36.462603 10346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:36.463171 10346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:36.464323 10346 hybrid_clock.cc:648] HybridClock initialized: now 1786342836464307 us; error 39 us; skew 500 ppm
Aug 10 06:20:36 dist-test-slave-q9h9 krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342836, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:36.466820 10346 init.cc:377] Logged in from keytab as oryx/127.31.204.193@KRBTEST.COM (short username oryx)
I20260810 06:20:36.467593 10346 webserver.cc:460] Webserver started at http://127.31.204.193:34127/ using document root <none> and password file <none>
I20260810 06:20:36.468070 10346 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:36.468133 10346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:36.468330 10346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:36.470148 10346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "e764aab3b1344241ac6bd8bc5639a237"
format_stamp: "Formatted at 2026-08-10 06:20:36 on dist-test-slave-q9h9"
server_key: "e3add201f1bff18c3ab31a5502dd1b5f"
server_key_iv: "910e6799138543dfefc4de1bd9ef84a2"
server_key_version: "encryptionkey@0"
I20260810 06:20:36.470659 10346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "e764aab3b1344241ac6bd8bc5639a237"
format_stamp: "Formatted at 2026-08-10 06:20:36 on dist-test-slave-q9h9"
server_key: "e3add201f1bff18c3ab31a5502dd1b5f"
server_key_iv: "910e6799138543dfefc4de1bd9ef84a2"
server_key_version: "encryptionkey@0"
I20260810 06:20:36.473905 10346 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 06:20:36.476051 10369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:36.477108 10346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:36.477208 10346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "e764aab3b1344241ac6bd8bc5639a237"
format_stamp: "Formatted at 2026-08-10 06:20:36 on dist-test-slave-q9h9"
server_key: "e3add201f1bff18c3ab31a5502dd1b5f"
server_key_iv: "910e6799138543dfefc4de1bd9ef84a2"
server_key_version: "encryptionkey@0"
I20260810 06:20:36.477300 10346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:36.494392 10346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:36.496711 10346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:36.496868 10346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:36.497357 10346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:36.498095 10346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:36.498142 10346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:36.498191 10346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:36.498220 10346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:36.506198 10346 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:41705
I20260810 06:20:36.506239 10572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:41705 every 8 connection(s)
I20260810 06:20:36.506938 10346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:20:36.513561 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 10346
I20260810 06:20:36.513692 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:20:36.514008 32563 external_mini_cluster.cc:1428] Setting key c987f82bdb95dba61099307f28f73175
Aug 10 06:20:36 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342836, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM
I20260810 06:20:36.519487 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:36.508476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:51263 (local address 127.31.204.254:34003)
0810 06:20:36.508588 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:36.508591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:36.509282 (+ 691us) server_negotiation.cc:407] Connection header received
0810 06:20:36.510597 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:36.510599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:36.510648 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:36.510714 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:36.511717 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:36.512233 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:36.513248 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:36.513459 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:36.513525 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:36.516242 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:36.516245 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:36.516247 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:36.516271 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:36.517526 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:36.518086 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:36.518089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:36.518091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:36.518165 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:36.518509 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:36.518511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:36.518512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:36.518662 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:36.518730 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:36.519216 (+ 486us) server_negotiation.cc:299] Negotiation successful
0810 06:20:36.519341 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:20:36.520107 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:36.508737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.193:51263)
0810 06:20:36.509148 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:36.509185 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:36.510395 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:36.510878 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:36.510884 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:36.511194 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:36.511602 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:36.511615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:36.512325 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:36.512327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:36.513154 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:36.513163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:36.513603 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:36.513608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:36.513809 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:36.514335 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:36.514354 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:36.516086 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:36.517641 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:36.517647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:36.517660 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:36.517972 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:36.518249 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:36.518252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:36.518254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:36.518424 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:36.518753 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:36.518758 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:36.518968 (+ 210us) client_negotiation.cc:769] Sending connection context
0810 06:20:36.519106 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 06:20:36.519328 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":110,"threads_started":1}
I20260810 06:20:36.521201 10574 heartbeater.cc:347] Connected to a master server at 127.31.204.254:34003
I20260810 06:20:36.521435 10574 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:36.521948 10574 heartbeater.cc:511] Master 127.31.204.254:34003 requested a full tablet report, sending...
I20260810 06:20:36.523391 10241 ts_manager.cc:194] Registered new tserver with Master: e764aab3b1344241ac6bd8bc5639a237 (127.31.204.193:41705)
WARNING: no policy specified for oryx/127.31.204.194@KRBTEST.COM; defaulting to no policy
I20260810 06:20:36.524503 10241 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:51263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.204.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.194.keytab.
Entry for principal oryx/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.194.keytab.
I20260810 06:20:36.539930 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34003
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.194.keytab
--principal=oryx/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:36.645213 10583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:36.645421 10583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:36.645458 10583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:36.649248 10583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:36.649294 10583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:36.649346 10583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:20:36.654161 10583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.194.keytab
--principal=oryx/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:36.655057 10583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:36.655934 10583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:36.664702 10594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:36.664772 10596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:36.664717 10590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:36.664964 10583 server_base.cc:1034] running on GCE node
I20260810 06:20:36.665405 10583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:36.665969 10583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:36.667162 10583 hybrid_clock.cc:648] HybridClock initialized: now 1786342836667138 us; error 31 us; skew 500 ppm
Aug 10 06:20:36 dist-test-slave-q9h9 krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342836, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:36.669715 10583 init.cc:377] Logged in from keytab as oryx/127.31.204.194@KRBTEST.COM (short username oryx)
I20260810 06:20:36.670559 10583 webserver.cc:460] Webserver started at http://127.31.204.194:42899/ using document root <none> and password file <none>
I20260810 06:20:36.671046 10583 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:36.671110 10583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:36.671301 10583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:36.673107 10583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "bc19037c918f422ba77c253d7c5eec17"
format_stamp: "Formatted at 2026-08-10 06:20:36 on dist-test-slave-q9h9"
server_key: "aa6c752d6b480e617e5b3742e005b964"
server_key_iv: "3e42cf15c23c96281452b71e625e2fd5"
server_key_version: "encryptionkey@0"
I20260810 06:20:36.673630 10583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "bc19037c918f422ba77c253d7c5eec17"
format_stamp: "Formatted at 2026-08-10 06:20:36 on dist-test-slave-q9h9"
server_key: "aa6c752d6b480e617e5b3742e005b964"
server_key_iv: "3e42cf15c23c96281452b71e625e2fd5"
server_key_version: "encryptionkey@0"
I20260810 06:20:36.677018 10583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:20:36.679180 10605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:36.680315 10583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:36.680415 10583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "bc19037c918f422ba77c253d7c5eec17"
format_stamp: "Formatted at 2026-08-10 06:20:36 on dist-test-slave-q9h9"
server_key: "aa6c752d6b480e617e5b3742e005b964"
server_key_iv: "3e42cf15c23c96281452b71e625e2fd5"
server_key_version: "encryptionkey@0"
I20260810 06:20:36.680503 10583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:36.712038 10583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:36.714510 10583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:36.714665 10583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:36.715204 10583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:36.715957 10583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:36.716003 10583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:36.716042 10583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:36.716070 10583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:36.723970 10583 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:46103
I20260810 06:20:36.723990 10799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:46103 every 8 connection(s)
I20260810 06:20:36.724743 10583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:20:36.725612 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 10583
I20260810 06:20:36.725731 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:20:36.726075 32563 external_mini_cluster.cc:1428] Setting key 80465f074162244b54711d68ca2f934e
Aug 10 06:20:36 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342836, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.31.204.195@KRBTEST.COM; defaulting to no policy
I20260810 06:20:36.738364 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:36.726528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:39055 (local address 127.31.204.254:34003)
0810 06:20:36.726671 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:36.726675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:36.727519 (+ 844us) server_negotiation.cc:407] Connection header received
0810 06:20:36.729077 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:36.729080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:36.729124 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:36.729192 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:36.730446 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:36.731177 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:36.732244 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:36.732590 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:36.732706 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:36.735224 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:36.735228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:36.735230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:36.735254 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:36.736555 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:36.737108 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:36.737111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:36.737112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:36.737190 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:36.737450 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:36.737452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:36.737453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:36.737630 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:36.737735 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:36.738146 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 06:20:36.738247 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 06:20:36.739109 10804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:36.726888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.194:39055)
0810 06:20:36.727400 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:36.727439 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:36.728865 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:36.729383 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:36.729391 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:36.729840 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:36.730307 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:36.730320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:36.731290 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:36.731292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:36.732111 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:36.732119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:36.732729 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:36.732732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:36.732869 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:36.733370 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:36.733390 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:36.735044 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:36.736679 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:36.736685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:36.736698 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:36.737023 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:36.737267 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:36.737269 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:36.737270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:36.737386 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:36.737725 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:36.737730 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:36.737933 (+ 203us) client_negotiation.cc:769] Sending connection context
0810 06:20:36.738108 (+ 175us) client_negotiation.cc:240] Negotiation successful
0810 06:20:36.738327 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":127,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.204.195@KRBTEST.COM" created.
I20260810 06:20:36.740129 10801 heartbeater.cc:347] Connected to a master server at 127.31.204.254:34003
I20260810 06:20:36.740335 10801 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:36.740790 10801 heartbeater.cc:511] Master 127.31.204.254:34003 requested a full tablet report, sending...
I20260810 06:20:36.741714 10241 ts_manager.cc:194] Registered new tserver with Master: bc19037c918f422ba77c253d7c5eec17 (127.31.204.194:46103)
I20260810 06:20:36.742317 10241 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:39055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.195.keytab.
Entry for principal oryx/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.195.keytab.
I20260810 06:20:36.752507 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34003
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.195.keytab
--principal=oryx/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:36.858270 10810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:36.858470 10810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:36.858508 10810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:36.862577 10810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:36.862625 10810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:36.862699 10810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:20:36.867684 10810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.195.keytab
--principal=oryx/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:36.868556 10810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:36.869418 10810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:36.877995 10824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:36.878151 10810 server_base.cc:1034] running on GCE node
W20260810 06:20:36.877972 10830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:36.877985 10822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:36.878533 10810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:36.879072 10810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:36.880221 10810 hybrid_clock.cc:648] HybridClock initialized: now 1786342836880223 us; error 43 us; skew 500 ppm
Aug 10 06:20:36 dist-test-slave-q9h9 krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342836, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:36.882721 10810 init.cc:377] Logged in from keytab as oryx/127.31.204.195@KRBTEST.COM (short username oryx)
I20260810 06:20:36.883486 10810 webserver.cc:460] Webserver started at http://127.31.204.195:41459/ using document root <none> and password file <none>
I20260810 06:20:36.883991 10810 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:36.884059 10810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:36.884277 10810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:36.886096 10810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "b59b0c145b014a6ea2444c33b301971c"
format_stamp: "Formatted at 2026-08-10 06:20:36 on dist-test-slave-q9h9"
server_key: "3b54804321c425d3e645acb0154f63bd"
server_key_iv: "a699813affcfd1e9978a303ee1ad66e5"
server_key_version: "encryptionkey@0"
I20260810 06:20:36.886607 10810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "b59b0c145b014a6ea2444c33b301971c"
format_stamp: "Formatted at 2026-08-10 06:20:36 on dist-test-slave-q9h9"
server_key: "3b54804321c425d3e645acb0154f63bd"
server_key_iv: "a699813affcfd1e9978a303ee1ad66e5"
server_key_version: "encryptionkey@0"
I20260810 06:20:36.889789 10810 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260810 06:20:36.891908 10840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:36.892946 10810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:36.893052 10810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "b59b0c145b014a6ea2444c33b301971c"
format_stamp: "Formatted at 2026-08-10 06:20:36 on dist-test-slave-q9h9"
server_key: "3b54804321c425d3e645acb0154f63bd"
server_key_iv: "a699813affcfd1e9978a303ee1ad66e5"
server_key_version: "encryptionkey@0"
I20260810 06:20:36.893136 10810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:36.920270 10810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:36.923058 10810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:36.923228 10810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:36.923729 10810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:36.924481 10810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:36.924528 10810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:36.924576 10810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:36.924606 10810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:36.932503 10810 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:43569
I20260810 06:20:36.932533 11024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:43569 every 8 connection(s)
I20260810 06:20:36.933265 10810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:20:36.937966 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 10810
I20260810 06:20:36.938093 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:20:36.938385 32563 external_mini_cluster.cc:1428] Setting key 117eaa690bee0ff9cc6f869a3f654997
Aug 10 06:20:36 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342836, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM
I20260810 06:20:36.945555 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:36.934934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:57253 (local address 127.31.204.254:34003)
0810 06:20:36.935059 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:36.935062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:36.935783 (+ 721us) server_negotiation.cc:407] Connection header received
0810 06:20:36.937031 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:36.937034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:36.937090 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:36.937161 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:36.938326 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:36.938918 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:36.940177 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:36.940418 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:36.940512 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:36.942700 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:36.942703 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:36.942705 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:36.942728 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:36.943987 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:36.944395 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:36.944398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:36.944399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:36.944475 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:36.944739 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:36.944741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:36.944742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:36.944896 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:36.944967 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:36.945347 (+ 380us) server_negotiation.cc:299] Negotiation successful
0810 06:20:36.945439 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:20:36.946323 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:36.935220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.195:57253)
0810 06:20:36.935637 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:36.935675 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:36.936816 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:36.937337 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:36.937343 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:36.937659 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:36.938159 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:36.938174 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:36.939038 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:36.939041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:36.940049 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:36.940060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:36.940553 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:36.940556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:36.940692 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:36.941117 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:36.941132 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:36.942555 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:36.944073 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:36.944077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:36.944086 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:36.944308 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:36.944546 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:36.944548 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:36.944549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:36.944661 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:36.944970 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:36.944974 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:36.945165 (+ 191us) client_negotiation.cc:769] Sending connection context
0810 06:20:36.945306 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 06:20:36.945517 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":106,"threads_started":1}
I20260810 06:20:36.947301 11025 heartbeater.cc:347] Connected to a master server at 127.31.204.254:34003
I20260810 06:20:36.947479 11025 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:36.947854 11025 heartbeater.cc:511] Master 127.31.204.254:34003 requested a full tablet report, sending...
I20260810 06:20:36.948743 10241 ts_manager.cc:194] Registered new tserver with Master: b59b0c145b014a6ea2444c33b301971c (127.31.204.195:43569)
I20260810 06:20:36.949278 10241 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:57253
I20260810 06:20:36.951889 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.204.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.204.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.204.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.253.keytab.
Entry for principal oryx/127.31.204.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.253.keytab.
I20260810 06:20:36.976504 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.253:36451
--webserver_interface=127.31.204.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.253.keytab
--principal=oryx/127.31.204.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:34003,127.31.204.253:36451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:37.081664 11031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:37.081911 11031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:37.081964 11031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:37.085887 11031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:37.085938 11031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:37.085964 11031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:37.085986 11031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:37.086037 11031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:20:37.090932 11031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:34003,127.31.204.253:36451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.253.keytab
--principal=oryx/127.31.204.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.253:36451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.204.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
W20260810 06:20:37.091117 11031 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.31.204.254:34003,127.31.204.253:36451'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 06:20:37.091846 11031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:37.092756 11031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:37.100430 11047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:37.100440 11042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:37.100668 11031 server_base.cc:1034] running on GCE node
W20260810 06:20:37.100442 11041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:37.101338 11031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:37.102356 11031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:37.103504 11031 hybrid_clock.cc:648] HybridClock initialized: now 1786342837103483 us; error 33 us; skew 500 ppm
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342837, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:37.105991 11031 init.cc:377] Logged in from keytab as oryx/127.31.204.253@KRBTEST.COM (short username oryx)
I20260810 06:20:37.106770 11031 webserver.cc:460] Webserver started at http://127.31.204.253:43327/ using document root <none> and password file <none>
I20260810 06:20:37.107221 11031 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:37.107273 11031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:37.107425 11031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:37.109261 11031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/data/instance:
uuid: "786f5f83aad243f5a9f6313c2c3681da"
format_stamp: "Formatted at 2026-08-10 06:20:37 on dist-test-slave-q9h9"
server_key: "e75032f6d3370399c662730642fc2691"
server_key_iv: "e80f9edf8038924a106473b674869350"
server_key_version: "encryptionkey@0"
I20260810 06:20:37.109823 11031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/wal/instance:
uuid: "786f5f83aad243f5a9f6313c2c3681da"
format_stamp: "Formatted at 2026-08-10 06:20:37 on dist-test-slave-q9h9"
server_key: "e75032f6d3370399c662730642fc2691"
server_key_iv: "e80f9edf8038924a106473b674869350"
server_key_version: "encryptionkey@0"
I20260810 06:20:37.113163 11031 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:20:37.115285 11061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:37.116351 11031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:37.116446 11031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/wal
uuid: "786f5f83aad243f5a9f6313c2c3681da"
format_stamp: "Formatted at 2026-08-10 06:20:37 on dist-test-slave-q9h9"
server_key: "e75032f6d3370399c662730642fc2691"
server_key_iv: "e80f9edf8038924a106473b674869350"
server_key_version: "encryptionkey@0"
I20260810 06:20:37.116518 11031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:37.132706 11031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:37.135692 11031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:37.135833 11031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:37.141923 11031 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.253:36451
I20260810 06:20:37.141938 11155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.253:36451 every 8 connection(s)
I20260810 06:20:37.142810 11031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/data/info.pb
I20260810 06:20:37.145259 11157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:37.148885 11157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:20:37.151364 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 11031
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342837, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.253@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.204.253@KRBTEST.COM
I20260810 06:20:37.161464 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.150509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40604 (local address 127.31.204.254:34003)
0810 06:20:37.150627 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.150631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.151421 (+ 790us) server_negotiation.cc:407] Connection header received
0810 06:20:37.152436 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.152440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.152498 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.152553 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.153680 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.154244 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.155225 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.155435 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.155523 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.157937 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.157941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.157943 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.157971 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.159346 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.159892 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.159896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.159898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.159973 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.160351 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.160354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.160355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.160643 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.160718 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.161202 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.161303 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:20:37.162286 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.152266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37224 (local address 127.31.204.253:36451)
0810 06:20:37.152608 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.152619 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.152657 (+ 38us) server_negotiation.cc:407] Connection header received
0810 06:20:37.152791 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.152795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.152940 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.153086 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.153894 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.154757 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.155600 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.155936 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.156026 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.158584 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.158590 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.158599 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.158619 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.160025 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.160451 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.160457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.160459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.160536 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.160784 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.160787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.160788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.160999 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.161146 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.161409 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.161731 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":255,"spinlock_wait_cycles":1920,"thread_start_us":160,"threads_started":1}
I20260810 06:20:37.162285 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.150818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:40604)
0810 06:20:37.151237 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.151302 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.152184 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.152815 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.152820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.153102 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:37.153536 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.153548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.154370 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.154373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.155101 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.155109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.155574 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.155578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.155735 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.156195 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:37.156210 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:37.157791 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:37.159457 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.159465 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.159474 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.159792 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.160063 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.160066 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.160068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.160239 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.160750 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:37.160756 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:37.160978 (+ 222us) client_negotiation.cc:769] Sending connection context
0810 06:20:37.161126 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 06:20:37.161409 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260810 06:20:37.164290 11157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342837, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.253@KRBTEST.COM for oryx/127.31.204.253@KRBTEST.COM
I20260810 06:20:37.171723 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.164642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:36451 (local address 127.0.0.1:37228)
0810 06:20:37.164763 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.164776 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.164857 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.165037 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.165040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.165174 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:37.165255 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.165260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.165913 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.165916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.167208 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.167219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.167645 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.167649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.167745 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.168078 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:37.168088 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:37.169162 (+ 1074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:37.170472 (+ 1310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.170476 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.170477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.170644 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.170889 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.170892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.170894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.170967 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.171287 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:37.171291 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:37.171367 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 06:20:37.171448 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 06:20:37.171566 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 06:20:37.171741 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.164706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.31.204.253:36451)
0810 06:20:37.164789 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.164793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.164820 (+ 27us) server_negotiation.cc:407] Connection header received
0810 06:20:37.164918 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.164921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.164954 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.165012 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.165356 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.165801 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.167312 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.167526 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.167608 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.169281 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.169284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.169286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.169305 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.170377 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.170739 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.170742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.170743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.170806 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.171045 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.171048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.171050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.171196 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.171276 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.171505 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.171620 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":35}
I20260810 06:20:37.174147 11157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: Bootstrap starting.
I20260810 06:20:37.176474 11157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:37.177107 11157 log.cc:824] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:37.178840 11157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: No bootstrap required, opened a new log
I20260810 06:20:37.181641 11157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } }
I20260810 06:20:37.181813 11157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:37.181859 11157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786f5f83aad243f5a9f6313c2c3681da, State: Initialized, Role: FOLLOWER
I20260810 06:20:37.182256 11157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } }
I20260810 06:20:37.182664 11166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } } }
I20260810 06:20:37.182771 11166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:20:37.183310 11157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [sys.catalog]: configured and running, proceeding with master startup.
W20260810 06:20:37.187559 11188 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:20:37.187637 11188 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:20:37.189582 11031 master_runner.cc:428] Detected that this master 786f5f83aad243f5a9f6313c2c3681da is joining an existing cluster
I20260810 06:20:37.189627 11031 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.204.253:36451
I20260810 06:20:37.199211 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.191400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:40606)
0810 06:20:37.191700 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.191712 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.191815 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.192061 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.192063 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.192207 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:37.192306 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.192311 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.193298 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.193300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.193879 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.193886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.194578 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.194582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.194675 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.195207 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:37.195221 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:37.195614 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:37.197559 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.197562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.197564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.197824 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.198111 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.198114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.198116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.198205 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.198668 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:37.198671 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:37.198757 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 06:20:37.198906 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 06:20:37.199029 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":20,"thread_start_us":81,"threads_started":1}
I20260810 06:20:37.199254 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.191465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40606 (local address 127.31.204.254:34003)
0810 06:20:37.191571 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.191575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.191799 (+ 224us) server_negotiation.cc:407] Connection header received
0810 06:20:37.191906 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.191909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.191957 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.192033 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.192408 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.193200 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.193986 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.194354 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.194453 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.195784 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.195788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.195790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.195816 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.197449 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.197905 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.197908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.197910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.198007 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.198319 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.198323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.198324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.198526 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.198596 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.198992 (+ 396us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.199099 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:20:37.200947 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.191481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:36451 (local address 127.0.0.1:37242)
0810 06:20:37.194172 (+ 2691us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.194182 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.194260 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.194444 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.194447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.194609 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:37.194726 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.194731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.195539 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.195542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.196302 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.196314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.196890 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.196894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.196990 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.197409 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:37.197423 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:37.197770 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:37.199437 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.199440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.199442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.199661 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.199945 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.199948 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.199950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.200044 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.200499 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:37.200502 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:37.200578 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 06:20:37.200660 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 06:20:37.200790 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2632,"mutex_wait_us":22,"thread_start_us":90,"threads_started":1}
I20260810 06:20:37.201220 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.191786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37242 (local address 127.31.204.253:36451)
0810 06:20:37.191877 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.191880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.194250 (+ 2370us) server_negotiation.cc:407] Connection header received
0810 06:20:37.194323 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.194326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.194360 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.194417 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.194847 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.195415 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.196429 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.196762 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.196870 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.197889 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.197893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.197895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.197917 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.199337 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.199753 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.199756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.199757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.199845 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.200251 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.200254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.200255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.200400 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.200484 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.200721 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.200845 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":40}
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342836, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.254@KRBTEST.COM for oryx/127.31.204.253@KRBTEST.COM
I20260810 06:20:37.210963 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.203260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37252 (local address 127.31.204.253:36451)
0810 06:20:37.203356 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.203359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.203613 (+ 254us) server_negotiation.cc:407] Connection header received
0810 06:20:37.203768 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.203771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.203816 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.203878 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.204438 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.204869 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.205678 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.205885 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.205966 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.208067 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.208071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.208073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.208097 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.209302 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.209798 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.209800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.209802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.209868 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.210166 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.210168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.210170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.210312 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.210394 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.210726 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.210833 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 06:20:37.210987 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.203219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:36451 (local address 127.0.0.1:37252)
0810 06:20:37.203504 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.203539 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.203654 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.203928 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.203934 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.204234 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:37.204325 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.204330 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.204971 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.204974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.205564 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.205572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.205984 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.205988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.206107 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.206529 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:37.206540 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:37.207941 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:37.209409 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.209419 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.209425 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.209707 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.209956 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.209959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.209961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.210089 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.210439 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:37.210446 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:37.210542 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 06:20:37.210653 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 06:20:37.210838 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":83,"threads_started":1}
I20260810 06:20:37.212054 10241 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.204.253:36451: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da"
member_type: NON_VOTER
last_known_addr {
host: "127.31.204.253"
port: 36451
}
attrs {
promote: true
}
}
dest_uuid: "cf988718c8da472c975d2fa2b1e2a341"
cas_config_opid_index: -1
I20260810 06:20:37.212597 10241 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: NON_VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: true } }
I20260810 06:20:37.213980 11208 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 786f5f83aad243f5a9f6313c2c3681da (127.31.204.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: NON_VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: true } } }
I20260810 06:20:37.214596 11111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:37.215658 11208 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.204.253:36451
I20260810 06:20:37.215764 11111 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 1 FOLLOWER]: Refusing update from remote peer cf988718c8da472c975d2fa2b1e2a341: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 06:20:37.215862 11208 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cf988718c8da472c975d2fa2b1e2a341" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: NON_VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: true } } }
I20260810 06:20:37.215946 11208 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:37.216337 11031 master.cc:537] Master@127.31.204.253:36451 shutting down...
I20260810 06:20:37.216308 11206 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: NON_VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 06:20:37.218398 10217 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 06:20:37.219231 10217 consensus_peers.cc:487] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 -> Peer 786f5f83aad243f5a9f6313c2c3681da (127.31.204.253:36451): Couldn't send request to peer 786f5f83aad243f5a9f6313c2c3681da. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 06:20:37.221421 11031 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:37.221642 11031 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:37.221728 11031 tablet_replica.cc:330] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: stopping tablet replica
WARNING: no policy specified for oryx/127.31.204.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.204.252@KRBTEST.COM" created.
I20260810 06:20:37.236408 11031 master.cc:552] Master@127.31.204.253:36451 shutdown complete.
I20260810 06:20:37.236456 11031 master_runner.cc:305] Clearing existing system tablet
I20260810 06:20:37.236680 11031 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:37.236735 11031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:37.237784 11031 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 06:20:37.239003 11220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:37.239243 11031 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:37.239353 11031 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 06:20:37.239389 11031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/wal
uuid: "786f5f83aad243f5a9f6313c2c3681da"
format_stamp: "Formatted at 2026-08-10 06:20:37 on dist-test-slave-q9h9"
server_key: "e75032f6d3370399c662730642fc2691"
server_key_iv: "e80f9edf8038924a106473b674869350"
server_key_version: "encryptionkey@0"
I20260810 06:20:37.241722 11031 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.204.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.252.keytab.
Entry for principal oryx/127.31.204.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.252.keytab.
I20260810 06:20:37.244407 11031 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 06:20:37.244498 11031 log.cc:1196] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 06:20:37.244746 11031 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: Deleting consensus metadata
I20260810 06:20:37.244910 11031 master_runner.cc:315] Copying system tablet from 127.31.204.254:34003
I20260810 06:20:37.244923 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.252:40965
--webserver_interface=127.31.204.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.252.keytab
--principal=oryx/127.31.204.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:34003,127.31.204.253:36451,127.31.204.252:40965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:20:37.246107 11031 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: tablet copy: Beginning tablet copy session from remote peer at address 127.31.204.254:34003
I20260810 06:20:37.255419 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.246793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40620 (local address 127.31.204.254:34003)
0810 06:20:37.246905 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.246909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.248105 (+ 1196us) server_negotiation.cc:407] Connection header received
0810 06:20:37.248206 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.248209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.248254 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.248318 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.248729 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.249242 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.250205 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.250507 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.250582 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.251957 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.251961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.251964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.251991 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.253469 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.253976 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.253980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.253982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.254111 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.254435 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.254438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.254440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.254651 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.254739 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.255149 (+ 410us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.255265 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:20:37.255946 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.246672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:40620)
0810 06:20:37.248002 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.248014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.248118 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.248353 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.248357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.248513 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:37.248614 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.248620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.249453 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.249457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.250069 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.250080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.250750 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.250753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.250846 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.251363 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:37.251380 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:37.251800 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:37.253596 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.253600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.253602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.253858 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.254233 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.254236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.254238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.254328 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.254841 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:37.254844 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:37.254948 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 06:20:37.255613 (+ 665us) client_negotiation.cc:240] Negotiation successful
0810 06:20:37.255769 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1269,"thread_start_us":84,"threads_started":1}
I20260810 06:20:37.256471 10273 tablet_copy_service.cc:140] P cf988718c8da472c975d2fa2b1e2a341: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 786f5f83aad243f5a9f6313c2c3681da ({username='oryx', principal='oryx/127.31.204.253@KRBTEST.COM'} at 127.0.0.1:40620)
I20260810 06:20:37.256644 10273 tablet_copy_service.cc:161] P cf988718c8da472c975d2fa2b1e2a341: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 786f5f83aad243f5a9f6313c2c3681da at {username='oryx', principal='oryx/127.31.204.253@KRBTEST.COM'} at 127.0.0.1:40620: session id = 786f5f83aad243f5a9f6313c2c3681da-00000000000000000000000000000000
I20260810 06:20:37.258447 10273 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 06:20:37.260044 11031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 06:20:37.262094 11031 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: tablet copy: Starting download of 0 data blocks...
I20260810 06:20:37.262269 11031 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: tablet copy: Starting download of 1 WAL segments...
I20260810 06:20:37.263684 11031 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 06:20:37.265229 10273 tablet_copy_service.cc:342] P cf988718c8da472c975d2fa2b1e2a341: Request end of tablet copy session 786f5f83aad243f5a9f6313c2c3681da-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.204.253@KRBTEST.COM'} at 127.0.0.1:40620
I20260810 06:20:37.265336 10273 tablet_copy_service.cc:433] P cf988718c8da472c975d2fa2b1e2a341: ending tablet copy session 786f5f83aad243f5a9f6313c2c3681da-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 786f5f83aad243f5a9f6313c2c3681da
I20260810 06:20:37.272218 11031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:37.273759 11232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:37.273983 11233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:37.274112 11235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:37.274328 11031 server_base.cc:1034] running on GCE node
I20260810 06:20:37.274645 11031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:37.274933 11031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:37.282259 11031 hybrid_clock.cc:648] HybridClock initialized: now 1786342837282232 us; error 44 us; skew 500 ppm
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342837, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:37.283826 11031 init.cc:377] Logged in from keytab as oryx/127.31.204.253@KRBTEST.COM (short username oryx)
I20260810 06:20:37.284288 11031 webserver.cc:460] Webserver started at http://127.31.204.253:39753/ using document root <none> and password file <none>
I20260810 06:20:37.284482 11031 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:37.284526 11031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:37.285810 11031 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:37.286811 11248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:37.287060 11031 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 06:20:37.287143 11031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/wal
uuid: "786f5f83aad243f5a9f6313c2c3681da"
format_stamp: "Formatted at 2026-08-10 06:20:37 on dist-test-slave-q9h9"
server_key: "e75032f6d3370399c662730642fc2691"
server_key_iv: "e80f9edf8038924a106473b674869350"
server_key_version: "encryptionkey@0"
I20260810 06:20:37.287218 11031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:37.324790 11031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:37.327004 11031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:37.331713 11031 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.253:36451
I20260810 06:20:37.331842 11338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.253:36451 every 8 connection(s)
I20260810 06:20:37.332280 11031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-1/data/info.pb
I20260810 06:20:37.334167 11339 sys_catalog.cc:263] Verifying existing consensus state
I20260810 06:20:37.335237 11339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: Bootstrap starting.
I20260810 06:20:37.346213 11339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:20:37.346506 11339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: Bootstrap complete.
I20260810 06:20:37.347065 11339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: NON_VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: true } }
I20260810 06:20:37.347200 11339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 786f5f83aad243f5a9f6313c2c3681da, State: Initialized, Role: LEARNER
I20260810 06:20:37.347313 11339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: NON_VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: true } }
I20260810 06:20:37.347642 11346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: NON_VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: true } } }
I20260810 06:20:37.347780 11339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:37.347852 11346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [sys.catalog]: This master's current role is: LEARNER
I20260810 06:20:37.350771 11364 catalog_manager.cc:1259] Loaded cluster ID: a7dcab4da1e8485897cdfde29453d3a9
I20260810 06:20:37.350888 11364 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: loading cluster ID for follower catalog manager: success
I20260810 06:20:37.352473 11364 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: acquiring CA information for follower catalog manager: success
I20260810 06:20:37.353543 11364 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 06:20:37.375100 11229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:37.375335 11229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:37.375393 11229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:37.379235 11229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:37.379281 11229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:37.379308 11229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:37.379330 11229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:37.379359 11229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:20:37.384523 11229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:34003,127.31.204.253:36451,127.31.204.252:40965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.252.keytab
--principal=oryx/127.31.204.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.252:40965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.204.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:37.385459 11229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:37.386433 11229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:37.394475 11373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:37.394511 11380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:37.394550 11229 server_base.cc:1034] running on GCE node
W20260810 06:20:37.394486 11374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:37.395231 11229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:37.396261 11229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:37.397440 11229 hybrid_clock.cc:648] HybridClock initialized: now 1786342837397433 us; error 66 us; skew 500 ppm
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342837, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:37.400008 11229 init.cc:377] Logged in from keytab as oryx/127.31.204.252@KRBTEST.COM (short username oryx)
I20260810 06:20:37.400804 11229 webserver.cc:460] Webserver started at http://127.31.204.252:45233/ using document root <none> and password file <none>
I20260810 06:20:37.401283 11229 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:37.401337 11229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:37.401494 11229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:37.403422 11229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/data/instance:
uuid: "db14022bd7e64ec4873973f0166a1fc8"
format_stamp: "Formatted at 2026-08-10 06:20:37 on dist-test-slave-q9h9"
server_key: "24e3704d0c3e2dd636ddb38c9d0a599c"
server_key_iv: "85b8dadc3f48d604eedaf59fb7d539b5"
server_key_version: "encryptionkey@0"
I20260810 06:20:37.403910 11229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/wal/instance:
uuid: "db14022bd7e64ec4873973f0166a1fc8"
format_stamp: "Formatted at 2026-08-10 06:20:37 on dist-test-slave-q9h9"
server_key: "24e3704d0c3e2dd636ddb38c9d0a599c"
server_key_iv: "85b8dadc3f48d604eedaf59fb7d539b5"
server_key_version: "encryptionkey@0"
I20260810 06:20:37.407383 11229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:20:37.409595 11389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:37.410790 11229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:37.410887 11229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/wal
uuid: "db14022bd7e64ec4873973f0166a1fc8"
format_stamp: "Formatted at 2026-08-10 06:20:37 on dist-test-slave-q9h9"
server_key: "24e3704d0c3e2dd636ddb38c9d0a599c"
server_key_iv: "85b8dadc3f48d604eedaf59fb7d539b5"
server_key_version: "encryptionkey@0"
I20260810 06:20:37.410979 11229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:37.442499 11229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:37.445158 11229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:37.445314 11229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:37.454152 11229 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.252:40965
I20260810 06:20:37.454388 11469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.252:40965 every 8 connection(s)
I20260810 06:20:37.455088 11229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/data/info.pb
I20260810 06:20:37.457713 11471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:37.461174 11471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:20:37.464856 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 11229
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342837, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.252@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.204.252@KRBTEST.COM
I20260810 06:20:37.474880 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.463115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40628 (local address 127.31.204.254:34003)
0810 06:20:37.463262 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.463267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.463865 (+ 598us) server_negotiation.cc:407] Connection header received
0810 06:20:37.464723 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.464726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.464779 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.464862 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.466225 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.466754 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.468213 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.468560 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.468639 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.471274 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.471277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.471279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.471302 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.472714 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.473327 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.473332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.473334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.473413 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.473767 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.473771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.473773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.473988 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.474115 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.474600 (+ 485us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.474701 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 06:20:37.475937 11477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.463226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:40628)
0810 06:20:37.463725 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.463765 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.464517 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.465052 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.465061 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.465479 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:37.466080 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.466101 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.466972 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.466977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.468076 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.468087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.468708 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.468713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.468940 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.469515 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:37.469532 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:37.471130 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:37.472829 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.472840 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.472849 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.473220 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.473516 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.473521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.473522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.473664 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.474109 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:37.474114 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:37.474341 (+ 227us) client_negotiation.cc:769] Sending connection context
0810 06:20:37.474559 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 06:20:37.474856 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":195,"threads_started":1}
I20260810 06:20:37.476123 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.465771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.31.204.252:40965)
0810 06:20:37.466201 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.466212 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.466233 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:20:37.466271 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.466274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.466408 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.466557 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.467484 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.468217 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.469327 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.469715 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.469798 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.472519 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.472526 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.472539 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.472565 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.474237 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.474733 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.474739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.474741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.474850 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.475151 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.475153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.475154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.475356 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.475508 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.475787 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.475920 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":196,"threads_started":1}
I20260810 06:20:37.477356 11471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342837, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.252@KRBTEST.COM for oryx/127.31.204.253@KRBTEST.COM
I20260810 06:20:37.485159 11477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.477756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:36451 (local address 127.0.0.1:37268)
0810 06:20:37.477869 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.477885 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.477960 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.478382 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.478385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.478533 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:37.478623 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.478629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.479523 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.479527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.480027 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.480035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.480534 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.480537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.480615 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.480944 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:37.480953 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:37.482094 (+ 1141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:37.483891 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.483894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.483895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.484059 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.484300 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.484302 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.484303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.484368 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.484782 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:37.484785 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:37.484868 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 06:20:37.484960 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 06:20:37.485056 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 06:20:37.485216 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.477889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.31.204.253:36451)
0810 06:20:37.478165 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.478169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.478181 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:20:37.478219 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.478221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.478260 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.478346 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.478749 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.479406 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.480165 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.480393 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.480522 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.482224 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.482227 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.482229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.482253 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.483801 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.484149 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.484152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.484153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.484229 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.484518 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.484521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.484522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.484690 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.484773 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.485000 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.485091 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":89,"threads_started":1}
I20260810 06:20:37.485881 11471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342837, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.252@KRBTEST.COM for oryx/127.31.204.252@KRBTEST.COM
I20260810 06:20:37.492515 11477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.486329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:40965 (local address 127.0.0.1:53884)
0810 06:20:37.486419 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.486434 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.486501 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.486665 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.486666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.486786 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:37.486870 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.486875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.487531 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.487533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.488009 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.488017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.488494 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.488499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.488568 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.488958 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:37.488971 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:37.490076 (+ 1105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:37.491404 (+ 1328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.491407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.491408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.491593 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.491837 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.491839 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.491840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.491907 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.492174 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:37.492175 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:37.492240 (+ 65us) client_negotiation.cc:769] Sending connection context
0810 06:20:37.492308 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 06:20:37.492415 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 06:20:37.492596 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.486381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.31.204.252:40965)
0810 06:20:37.486474 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.486477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.486488 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:20:37.486567 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.486568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.486599 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.486654 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.486955 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.487437 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.488111 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.488379 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.488463 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.490205 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.490208 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.490209 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.490228 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.491279 (+ 1051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.491685 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.491688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.491689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.491757 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.491970 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.491971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.491973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.492105 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.492179 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.492351 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.492467 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":37}
I20260810 06:20:37.495487 11471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: Bootstrap starting.
I20260810 06:20:37.497776 11471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:37.498504 11471 log.cc:824] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:37.500192 11471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: No bootstrap required, opened a new log
I20260810 06:20:37.503072 11471 raft_consensus.cc:348] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } }
I20260810 06:20:37.503255 11471 raft_consensus.cc:374] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:37.503311 11471 raft_consensus.cc:729] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db14022bd7e64ec4873973f0166a1fc8, State: Initialized, Role: FOLLOWER
I20260810 06:20:37.503683 11471 consensus_queue.cc:260] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } }
I20260810 06:20:37.504113 11482 sys_catalog.cc:455] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } } }
I20260810 06:20:37.504221 11482 sys_catalog.cc:458] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:20:37.504751 11471 sys_catalog.cc:564] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 06:20:37.509145 11505 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:20:37.509274 11505 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:20:37.512126 11229 master_runner.cc:428] Detected that this master db14022bd7e64ec4873973f0166a1fc8 is joining an existing cluster
I20260810 06:20:37.512172 11229 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.204.252:40965
I20260810 06:20:37.522785 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.514731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.31.204.253:36451)
0810 06:20:37.514872 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.514876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.515441 (+ 565us) server_negotiation.cc:407] Connection header received
0810 06:20:37.515609 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.515613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.515661 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.515734 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.516315 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.517093 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.518382 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.518718 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.518807 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.520028 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.520031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.520033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.520053 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.521313 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.521758 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.521761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.521762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.521829 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.522099 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.522101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.522102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.522254 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.522326 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.522587 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.522680 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 06:20:37.522920 11519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.514508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:36451 (local address 127.0.0.1:37280)
0810 06:20:37.515367 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.515378 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.515501 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.515858 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.515861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.516060 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:37.516179 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.516186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.517231 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.517234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.518250 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.518260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.518952 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.518956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.519045 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.519445 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:37.519460 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:37.519897 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:37.521419 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.521422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.521424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.521673 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.521906 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.521909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.521911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.521999 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.522342 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:37.522344 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:37.522442 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 06:20:37.522543 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 06:20:37.522664 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":51,"thread_start_us":343,"threads_started":1}
I20260810 06:20:37.526597 10574 heartbeater.cc:503] Master 127.31.204.254:34003 was elected leader, sending a full tablet report...
I20260810 06:20:37.529069 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 10346
I20260810 06:20:37.542475 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.514911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53898 (local address 127.31.204.252:40965)
0810 06:20:37.515002 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.515007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.515798 (+ 791us) server_negotiation.cc:407] Connection header received
0810 06:20:37.515917 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.515920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.515951 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.515989 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.516506 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.517182 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.518880 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.519176 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.519252 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.539223 (+ 19971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.539227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.539230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.539254 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.540714 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.541223 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.541226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.541228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.541304 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.541563 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.541565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.541567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.541767 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.541839 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.542224 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.542331 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 06:20:37.542486 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.514524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40638 (local address 127.31.204.254:34003)
0810 06:20:37.514753 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.514758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.515090 (+ 332us) server_negotiation.cc:407] Connection header received
0810 06:20:37.515230 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.515234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.515281 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.515359 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.515861 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.516618 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.517761 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.518148 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.518223 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.538844 (+ 20621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.538849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.538852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.538879 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.540782 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.541250 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.541253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.541254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.541331 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.541595 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.541598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.541600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.541780 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.541840 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.542248 (+ 408us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.542349 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260810 06:20:37.542915 11515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.514484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:40638)
0810 06:20:37.515013 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.515026 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.515142 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.515392 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.515395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.515599 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:37.515737 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.515745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.516739 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.516742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.517624 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.517635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.518379 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.518383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.518468 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.518934 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:37.518947 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:37.538680 (+ 19733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:37.540882 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.540884 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.540886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.541151 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.541430 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.541432 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.541434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.541501 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.541900 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:37.541902 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:37.542051 (+ 149us) client_negotiation.cc:769] Sending connection context
0810 06:20:37.542605 (+ 554us) client_negotiation.cc:240] Negotiation successful
0810 06:20:37.542755 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":29,"spinlock_wait_cycles":2688,"thread_start_us":660,"threads_started":1}
I20260810 06:20:37.543320 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.515088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:40965 (local address 127.0.0.1:53898)
0810 06:20:37.515725 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.515737 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.515845 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.516081 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.516084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.516280 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:37.516407 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.516414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.517285 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.517288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.518767 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.518778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.519376 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.519379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.519465 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.538697 (+ 19232us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:37.538711 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:37.539096 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:37.540824 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.540827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.540829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.541127 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.541398 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.541401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.541402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.541477 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.541898 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:37.541901 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:37.542046 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 06:20:37.543025 (+ 979us) client_negotiation.cc:240] Negotiation successful
0810 06:20:37.543164 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":60,"spinlock_wait_cycles":3584,"thread_start_us":121,"threads_started":1}
I20260810 06:20:37.545315 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:41705
--local_ip_for_outbound_sockets=127.31.204.193
--tserver_master_addrs=127.31.204.254:34003
--webserver_port=34127
--webserver_interface=127.31.204.193
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.193.keytab
--principal=oryx/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.204.254:34003,127.31.204.253:36451
--tserver_master_addrs=127.31.204.254:34003,127.31.204.253:36451,127.31.204.252:40965 with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342836, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.254@KRBTEST.COM for oryx/127.31.204.252@KRBTEST.COM
I20260810 06:20:37.554478 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.547195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53912 (local address 127.31.204.252:40965)
0810 06:20:37.547287 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.547291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.547306 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:20:37.547472 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.547475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.547522 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.547608 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.548047 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.548659 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.549521 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.549881 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.549967 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.551761 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.551765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.551768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.551790 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.552929 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.553338 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.553341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.553343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.553412 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.553668 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.553671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.553673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.553823 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.553903 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.554257 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.554347 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 06:20:37.554489 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.547109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:40965 (local address 127.0.0.1:53912)
0810 06:20:37.547261 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.547274 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.547338 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.547631 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.547634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.547816 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:37.547935 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.547942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.548781 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.548783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.549411 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.549420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.549993 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.549996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.550113 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.550472 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:37.550481 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:37.551634 (+ 1153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:37.553034 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.553038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.553040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.553244 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.553500 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.553503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.553505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.553583 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.553919 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:37.553922 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:37.554005 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 06:20:37.554205 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 06:20:37.554339 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260810 06:20:37.555600 10241 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.204.252:40965: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8"
member_type: NON_VOTER
last_known_addr {
host: "127.31.204.252"
port: 40965
}
attrs {
promote: true
}
}
dest_uuid: "cf988718c8da472c975d2fa2b1e2a341"
cas_config_opid_index: 5
I20260810 06:20:37.556123 10241 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: NON_VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: true } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: NON_VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: true } }
I20260810 06:20:37.557129 11206 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER db14022bd7e64ec4873973f0166a1fc8 (127.31.204.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: NON_VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: true } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: NON_VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: true } } }
I20260810 06:20:37.557631 11431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:37.559028 11431 raft_consensus.cc:1261] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 1 FOLLOWER]: Refusing update from remote peer cf988718c8da472c975d2fa2b1e2a341: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 06:20:37.559114 11206 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.204.252:40965
I20260810 06:20:37.559268 11206 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cf988718c8da472c975d2fa2b1e2a341" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: NON_VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: true } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: NON_VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: true } } }
I20260810 06:20:37.559360 11206 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:37.559633 11229 master.cc:537] Master@127.31.204.252:40965 shutting down...
I20260810 06:20:37.560298 11208 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: NON_VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 06:20:37.561950 10219 consensus_peers.cc:487] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 -> Peer db14022bd7e64ec4873973f0166a1fc8 (127.31.204.252:40965): Couldn't send request to peer db14022bd7e64ec4873973f0166a1fc8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 06:20:37.565719 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.558247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:36451 (local address 127.0.0.1:37286)
0810 06:20:37.559508 (+ 1261us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.559521 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.561940 (+ 2419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.562385 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.562388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.562407 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:37.562483 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.562488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.563433 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.563436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.564368 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.564379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.565339 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.565343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.565424 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.565471 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 06:20:37.565540 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 06:20:37.565604 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1200}
I20260810 06:20:37.566176 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.558458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.31.204.253:36451)
0810 06:20:37.560194 (+ 1736us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.560199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.560213 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:20:37.562107 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.562111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.562174 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.562261 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:37.562605 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.563311 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.564514 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.565028 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.565147 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.565276 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.565906 (+ 630us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.566056 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1656}
I20260810 06:20:37.566679 11289 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 1 LEARNER]: Refusing update from remote peer cf988718c8da472c975d2fa2b1e2a341: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 06:20:37.567142 11229 raft_consensus.cc:2229] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:37.567145 11208 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: NON_VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 06:20:37.567411 11229 raft_consensus.cc:2258] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 06:20:37.567550 11482 raft_consensus.cc:659] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopped
I20260810 06:20:37.567782 11229 tablet_replica.cc:330] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: stopping tablet replica
I20260810 06:20:37.569001 11289 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER db14022bd7e64ec4873973f0166a1fc8 (127.31.204.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: NON_VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: true } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: NON_VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: true } } }
I20260810 06:20:37.570564 11525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf988718c8da472c975d2fa2b1e2a341. Latest consensus state: current_term: 1 leader_uuid: "cf988718c8da472c975d2fa2b1e2a341" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: NON_VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: true } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: NON_VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: true } } }
I20260810 06:20:37.570542 11346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cf988718c8da472c975d2fa2b1e2a341" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: NON_VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: true } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: NON_VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: true } } }
I20260810 06:20:37.570672 11346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [sys.catalog]: This master's current role is: LEARNER
I20260810 06:20:37.570737 11525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [sys.catalog]: This master's current role is: LEARNER
I20260810 06:20:37.572449 11208 raft_consensus.cc:1050] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341: attempting to promote NON_VOTER 786f5f83aad243f5a9f6313c2c3681da to VOTER
I20260810 06:20:37.572880 11208 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: NON_VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: true } }
I20260810 06:20:37.574110 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.573796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:40965 (local address 127.0.0.1:53916)
0810 06:20:37.573927 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.573994 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:40965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 06:20:37.574389 11289 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 1 LEARNER]: Refusing update from remote peer cf988718c8da472c975d2fa2b1e2a341: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 06:20:37.574755 11206 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 06:20:37.574960 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.574715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:40965 (local address 127.0.0.1:53922)
0810 06:20:37.574857 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.574904 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:40965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260810 06:20:37.575131 10219 consensus_peers.cc:487] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 -> Peer db14022bd7e64ec4873973f0166a1fc8 (127.31.204.252:40965): Couldn't send request to peer db14022bd7e64ec4873973f0166a1fc8. Status: Network error: Client connection negotiation failed: client connection to 127.31.204.252:40965: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 06:20:37.576215 11206 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 786f5f83aad243f5a9f6313c2c3681da (127.31.204.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: NON_VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: true } } }
I20260810 06:20:37.577277 11206 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cf988718c8da472c975d2fa2b1e2a341" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: NON_VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: true } } }
I20260810 06:20:37.577378 11206 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:37.577510 11206 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cf988718c8da472c975d2fa2b1e2a341" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: NON_VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: true } } }
I20260810 06:20:37.577571 11206 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:37.578512 11289 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 786f5f83aad243f5a9f6313c2c3681da (127.31.204.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: NON_VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: true } } }
I20260810 06:20:37.579638 11525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cf988718c8da472c975d2fa2b1e2a341" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: NON_VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: true } } }
I20260810 06:20:37.579737 11525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:20:37.584497 11229 master.cc:552] Master@127.31.204.252:40965 shutdown complete.
I20260810 06:20:37.584542 11229 master_runner.cc:305] Clearing existing system tablet
I20260810 06:20:37.584782 11229 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:37.584841 11229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:37.585830 11229 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 06:20:37.586642 11529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:37.586862 11229 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:37.586984 11229 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 06:20:37.587024 11229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/wal
uuid: "db14022bd7e64ec4873973f0166a1fc8"
format_stamp: "Formatted at 2026-08-10 06:20:37 on dist-test-slave-q9h9"
server_key: "24e3704d0c3e2dd636ddb38c9d0a599c"
server_key_iv: "85b8dadc3f48d604eedaf59fb7d539b5"
server_key_version: "encryptionkey@0"
I20260810 06:20:37.588826 11229 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:37.590664 11229 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 06:20:37.590750 11229 log.cc:1196] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 06:20:37.590974 11229 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: Deleting consensus metadata
I20260810 06:20:37.591116 11229 master_runner.cc:315] Copying system tablet from 127.31.204.254:34003
I20260810 06:20:37.592069 11229 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: tablet copy: Beginning tablet copy session from remote peer at address 127.31.204.254:34003
I20260810 06:20:37.601104 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.594414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40650 (local address 127.31.204.254:34003)
0810 06:20:37.594536 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.594541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.594751 (+ 210us) server_negotiation.cc:407] Connection header received
0810 06:20:37.594882 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.594885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.594929 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.594992 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.595425 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.596014 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.596911 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.597124 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.597218 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.598317 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.598320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.598323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.598347 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.599601 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.599993 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.599996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.599997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.600069 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.600328 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.600331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.600333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.600494 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.600576 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.600854 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.600953 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 06:20:37.601114 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.594345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:40650)
0810 06:20:37.594663 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.594675 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.594777 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.595025 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.595028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.595211 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:37.595310 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.595316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.596117 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.596121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.596799 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.596811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.597237 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.597241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.597337 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.597817 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:37.597832 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:37.598193 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:37.599709 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.599712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.599715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.599898 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.600157 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.600160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.600162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.600243 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.600590 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:37.600593 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:37.600698 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 06:20:37.600798 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 06:20:37.600927 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":90,"threads_started":1}
I20260810 06:20:37.601794 10273 tablet_copy_service.cc:140] P cf988718c8da472c975d2fa2b1e2a341: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer db14022bd7e64ec4873973f0166a1fc8 ({username='oryx', principal='oryx/127.31.204.252@KRBTEST.COM'} at 127.0.0.1:40650)
I20260810 06:20:37.601877 10273 tablet_copy_service.cc:161] P cf988718c8da472c975d2fa2b1e2a341: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer db14022bd7e64ec4873973f0166a1fc8 at {username='oryx', principal='oryx/127.31.204.252@KRBTEST.COM'} at 127.0.0.1:40650: session id = db14022bd7e64ec4873973f0166a1fc8-00000000000000000000000000000000
I20260810 06:20:37.602699 10273 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 06:20:37.603696 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 06:20:37.605669 11229 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: tablet copy: Starting download of 0 data blocks...
I20260810 06:20:37.605871 11229 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: tablet copy: Starting download of 1 WAL segments...
I20260810 06:20:37.607291 11229 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 06:20:37.608891 10273 tablet_copy_service.cc:342] P cf988718c8da472c975d2fa2b1e2a341: Request end of tablet copy session db14022bd7e64ec4873973f0166a1fc8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.204.252@KRBTEST.COM'} at 127.0.0.1:40650
I20260810 06:20:37.608983 10273 tablet_copy_service.cc:433] P cf988718c8da472c975d2fa2b1e2a341: ending tablet copy session db14022bd7e64ec4873973f0166a1fc8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer db14022bd7e64ec4873973f0166a1fc8
I20260810 06:20:37.615697 11229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:37.617455 11547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:37.617452 11542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:37.617573 11544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:37.617825 11229 server_base.cc:1034] running on GCE node
I20260810 06:20:37.617985 11229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:37.618279 11229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:37.619421 11229 hybrid_clock.cc:648] HybridClock initialized: now 1786342837619412 us; error 24 us; skew 500 ppm
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342837, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:37.620723 11229 init.cc:377] Logged in from keytab as oryx/127.31.204.252@KRBTEST.COM (short username oryx)
I20260810 06:20:37.621165 11229 webserver.cc:460] Webserver started at http://127.31.204.252:40827/ using document root <none> and password file <none>
I20260810 06:20:37.621338 11229 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:37.621374 11229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:37.622409 11229 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 06:20:37.623252 11564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:37.623456 11229 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 06:20:37.623525 11229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/wal
uuid: "db14022bd7e64ec4873973f0166a1fc8"
format_stamp: "Formatted at 2026-08-10 06:20:37 on dist-test-slave-q9h9"
server_key: "24e3704d0c3e2dd636ddb38c9d0a599c"
server_key_iv: "85b8dadc3f48d604eedaf59fb7d539b5"
server_key_version: "encryptionkey@0"
I20260810 06:20:37.623606 11229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:37.641498 11229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:37.643440 11229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:37.648468 11229 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.252:40965
I20260810 06:20:37.649156 11229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/master-2/data/info.pb
I20260810 06:20:37.649852 11650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.252:40965 every 8 connection(s)
I20260810 06:20:37.651413 11652 sys_catalog.cc:263] Verifying existing consensus state
I20260810 06:20:37.652559 11652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: Bootstrap starting.
I20260810 06:20:37.668962 11652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:20:37.669231 11652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: Bootstrap complete.
I20260810 06:20:37.669699 11652 raft_consensus.cc:348] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: NON_VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: true } }
I20260810 06:20:37.669813 11652 raft_consensus.cc:729] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: db14022bd7e64ec4873973f0166a1fc8, State: Initialized, Role: LEARNER
I20260810 06:20:37.669910 11652 consensus_queue.cc:260] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: NON_VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: true } }
I20260810 06:20:37.670241 11656 sys_catalog.cc:455] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: NON_VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: true } } }
I20260810 06:20:37.670413 11652 sys_catalog.cc:564] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:37.670418 11656 sys_catalog.cc:458] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [sys.catalog]: This master's current role is: LEARNER
I20260810 06:20:37.673380 11671 catalog_manager.cc:1259] Loaded cluster ID: a7dcab4da1e8485897cdfde29453d3a9
I20260810 06:20:37.673488 11671 catalog_manager.cc:1552] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: loading cluster ID for follower catalog manager: success
I20260810 06:20:37.675254 11671 catalog_manager.cc:1574] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: acquiring CA information for follower catalog manager: success
I20260810 06:20:37.676335 11671 catalog_manager.cc:1602] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 06:20:37.703784 11522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:37.704003 11522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:37.704043 11522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:37.707875 11522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:37.707918 11522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:37.707976 11522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:20:37.713011 11522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.193.keytab
--principal=oryx/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:41705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=34127
--tserver_master_addrs=127.31.204.254:34003,127.31.204.253:36451,127.31.204.252:40965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:37.713945 11522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:37.714939 11522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:37.723693 11682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:37.723740 11686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:37.724025 11522 server_base.cc:1034] running on GCE node
W20260810 06:20:37.724164 11683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:37.724546 11522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:37.725133 11522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:37.726305 11522 hybrid_clock.cc:648] HybridClock initialized: now 1786342837726285 us; error 29 us; skew 500 ppm
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342837, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:37.728868 11522 init.cc:377] Logged in from keytab as oryx/127.31.204.193@KRBTEST.COM (short username oryx)
I20260810 06:20:37.729593 11522 webserver.cc:460] Webserver started at http://127.31.204.193:34127/ using document root <none> and password file <none>
I20260810 06:20:37.730113 11522 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:37.730191 11522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:37.733515 11522 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:37.735383 11696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:37.736445 11522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:37.736537 11522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "e764aab3b1344241ac6bd8bc5639a237"
format_stamp: "Formatted at 2026-08-10 06:20:36 on dist-test-slave-q9h9"
server_key: "e3add201f1bff18c3ab31a5502dd1b5f"
server_key_iv: "910e6799138543dfefc4de1bd9ef84a2"
server_key_version: "encryptionkey@0"
I20260810 06:20:37.737010 11522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:37.744153 10801 heartbeater.cc:503] Master 127.31.204.254:34003 was elected leader, sending a full tablet report...
I20260810 06:20:37.761453 11522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:37.764183 11522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:37.764317 11522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:37.764792 11522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:37.765513 11522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:37.765558 11522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:37.765592 11522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:37.765612 11522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:37.776268 11522 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:41705
I20260810 06:20:37.776291 11874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:41705 every 8 connection(s)
I20260810 06:20:37.777158 11522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:20:37.778921 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 11522
I20260810 06:20:37.779109 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 10583
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342837, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.253@KRBTEST.COM
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342837, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM
I20260810 06:20:37.790606 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:46103
--local_ip_for_outbound_sockets=127.31.204.194
--tserver_master_addrs=127.31.204.254:34003
--webserver_port=42899
--webserver_interface=127.31.204.194
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.194.keytab
--principal=oryx/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.204.254:34003,127.31.204.253:36451
--tserver_master_addrs=127.31.204.254:34003,127.31.204.253:36451,127.31.204.252:40965 with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342837, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.252@KRBTEST.COM
I20260810 06:20:37.794485 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.782862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:56523 (local address 127.31.204.253:36451)
0810 06:20:37.783249 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.783253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.783266 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:20:37.783307 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.783310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.783355 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.783434 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.784653 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.785167 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.786389 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.786731 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.786819 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.790477 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.790481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.790483 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.790505 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.792107 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.792748 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.792752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.792754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.792843 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.793332 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.793336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.793338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.793598 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.793696 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.794210 (+ 514us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.794329 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300}
I20260810 06:20:37.795212 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.782720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:56643 (local address 127.31.204.254:34003)
0810 06:20:37.782847 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.782852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.782868 (+ 16us) server_negotiation.cc:407] Connection header received
0810 06:20:37.783014 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.783017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.783071 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.783155 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.784667 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.785421 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.786588 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.786902 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.786972 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.790881 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.790886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.790889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.790918 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.792589 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.793025 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.793029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.793030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.793119 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.793416 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.793419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.793421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.793608 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.793696 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.794947 (+ 1251us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.795061 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:20:37.795397 11885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.780330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:36451 (local address 127.31.204.193:56523)
0810 06:20:37.782507 (+ 2177us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.782520 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.783075 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.783474 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.783477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.784007 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:37.784497 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.784509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.785309 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.785311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.786241 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.786252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.787109 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.787113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.787296 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.787947 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:37.787965 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:37.790316 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:37.792218 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.792224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.792238 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.792563 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.792995 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.792998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.792999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.793220 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.793723 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:37.793729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:37.793960 (+ 231us) client_negotiation.cc:769] Sending connection context
0810 06:20:37.794129 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 06:20:37.794514 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2116,"mutex_wait_us":131,"spinlock_wait_cycles":7168,"thread_start_us":1228,"threads_started":1}
I20260810 06:20:37.795666 11882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.779275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.193:56643)
0810 06:20:37.781824 (+ 2549us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.781875 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.782807 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.783445 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.783453 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.784007 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:37.784499 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.784512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.785561 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.785564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.786456 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.786467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.787156 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.787160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.787300 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.788121 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:37.788174 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:37.790715 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:37.792682 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.792685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.792687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.792921 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.793231 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.793234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.793236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.793316 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.793756 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:37.793758 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:37.793945 (+ 187us) client_negotiation.cc:769] Sending connection context
0810 06:20:37.794129 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 06:20:37.794456 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2337,"mutex_wait_us":31,"spinlock_wait_cycles":5888,"thread_start_us":2209,"threads_started":1}
I20260810 06:20:37.797063 11877 heartbeater.cc:347] Connected to a master server at 127.31.204.253:36451
I20260810 06:20:37.797235 11877 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:37.797745 11877 heartbeater.cc:511] Master 127.31.204.253:36451 requested a full tablet report, sending...
I20260810 06:20:37.798517 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.782238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:40965 (local address 127.31.204.193:46435)
0810 06:20:37.786425 (+ 4187us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.786438 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.786509 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.787002 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.787005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.787182 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:37.787299 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.787305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.788478 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.788481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.789277 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.789289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.790006 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.790010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.790117 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.790765 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:37.790780 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:37.793167 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:37.796892 (+ 3725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.796896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.796897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.797135 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.797496 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.797499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.797501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.797584 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.798069 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:37.798072 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:37.798157 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 06:20:37.798252 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 06:20:37.798373 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4127,"thread_start_us":119,"threads_started":1}
I20260810 06:20:37.798980 11886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.782992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:46435 (local address 127.31.204.252:40965)
0810 06:20:37.786693 (+ 3701us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.786706 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.786719 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:20:37.786757 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.786760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.786807 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.786881 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.787434 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.788350 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.789412 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.789773 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.789871 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.793327 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.793331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.793333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.793359 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.796801 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.797285 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.797288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.797290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.797383 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.797701 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.797704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.797706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.797911 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.798002 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.798707 (+ 705us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.798833 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3627,"thread_start_us":93,"threads_started":1}
I20260810 06:20:37.799443 11273 ts_manager.cc:194] Registered new tserver with Master: e764aab3b1344241ac6bd8bc5639a237 (127.31.204.193:41705)
I20260810 06:20:37.800509 11876 heartbeater.cc:347] Connected to a master server at 127.31.204.254:34003
I20260810 06:20:37.800626 11876 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:37.800637 11875 heartbeater.cc:347] Connected to a master server at 127.31.204.252:40965
I20260810 06:20:37.800776 11875 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:37.800880 11876 heartbeater.cc:511] Master 127.31.204.254:34003 requested a full tablet report, sending...
I20260810 06:20:37.800956 11875 heartbeater.cc:511] Master 127.31.204.252:40965 requested a full tablet report, sending...
I20260810 06:20:37.801628 10241 ts_manager.cc:194] Re-registered known tserver with Master: e764aab3b1344241ac6bd8bc5639a237 (127.31.204.193:41705)
I20260810 06:20:37.802188 11590 ts_manager.cc:194] Registered new tserver with Master: e764aab3b1344241ac6bd8bc5639a237 (127.31.204.193:41705)
I20260810 06:20:37.802358 10241 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:56643
W20260810 06:20:37.899747 11888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:37.899961 11888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:37.900015 11888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:37.903913 11888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:37.903963 11888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:37.904042 11888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:20:37.908885 11888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.194.keytab
--principal=oryx/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:46103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=42899
--tserver_master_addrs=127.31.204.254:34003,127.31.204.253:36451,127.31.204.252:40965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:37.909749 11888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:37.910758 11888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:37.919587 11900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:37.919763 11888 server_base.cc:1034] running on GCE node
W20260810 06:20:37.919569 11895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:37.919575 11894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:37.920231 11888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:37.920753 11888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:37.921919 11888 hybrid_clock.cc:648] HybridClock initialized: now 1786342837921906 us; error 25 us; skew 500 ppm
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342837, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:37.924417 11888 init.cc:377] Logged in from keytab as oryx/127.31.204.194@KRBTEST.COM (short username oryx)
I20260810 06:20:37.925205 11888 webserver.cc:460] Webserver started at http://127.31.204.194:42899/ using document root <none> and password file <none>
I20260810 06:20:37.925670 11888 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:37.925738 11888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:37.929354 11888 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:37.931248 11913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:37.932318 11888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:37.932428 11888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "bc19037c918f422ba77c253d7c5eec17"
format_stamp: "Formatted at 2026-08-10 06:20:36 on dist-test-slave-q9h9"
server_key: "aa6c752d6b480e617e5b3742e005b964"
server_key_iv: "3e42cf15c23c96281452b71e625e2fd5"
server_key_version: "encryptionkey@0"
I20260810 06:20:37.932900 11888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:37.951038 11025 heartbeater.cc:503] Master 127.31.204.254:34003 was elected leader, sending a full tablet report...
I20260810 06:20:37.956879 11888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:37.959319 11888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:37.959475 11888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:37.959998 11888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:37.960745 11888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:37.960796 11888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:37.960877 11888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:37.960907 11888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:37.972870 11888 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:46103
I20260810 06:20:37.972906 12096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:46103 every 8 connection(s)
I20260810 06:20:37.973735 11888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:20:37.978401 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 11888
I20260810 06:20:37.978578 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 10810
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342837, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.253@KRBTEST.COM
I20260810 06:20:37.991731 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.976073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:41371 (local address 127.31.204.253:36451)
0810 06:20:37.976188 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.976194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.977246 (+ 1052us) server_negotiation.cc:407] Connection header received
0810 06:20:37.977749 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.977752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.977810 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.977861 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.983401 (+ 5540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.983912 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.985153 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.985428 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.985506 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.988386 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:37.988389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:37.988392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:37.988416 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:37.989753 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.990312 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.990316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.990318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.990394 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:37.990698 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:37.990701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:37.990702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:37.990903 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:37.990969 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:37.991482 (+ 513us) server_negotiation.cc:299] Negotiation successful
0810 06:20:37.991580 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:20:37.993104 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.975977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:36451 (local address 127.31.204.194:41371)
0810 06:20:37.977143 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.977156 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.977545 (+ 389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.978085 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.978094 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.982828 (+ 4734us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:37.983262 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.983275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.984037 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.984039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.985031 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.985041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.985741 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.985748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.986012 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.986683 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:37.986700 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:37.988223 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:37.989888 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.989892 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.989902 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.990203 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.990483 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:37.990485 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:37.990487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:37.990619 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:37.991027 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:37.991032 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:37.991260 (+ 228us) client_negotiation.cc:769] Sending connection context
0810 06:20:37.992022 (+ 762us) client_negotiation.cc:240] Negotiation successful
0810 06:20:37.992284 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1086,"mutex_wait_us":856,"thread_start_us":107,"threads_started":1}
Aug 10 06:20:37 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342837, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM
I20260810 06:20:38.004556 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.975569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:37693 (local address 127.31.204.254:34003)
0810 06:20:37.975689 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.975693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.976675 (+ 982us) server_negotiation.cc:407] Connection header received
0810 06:20:37.977748 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.977752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.977808 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.977861 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:37.995825 (+ 17964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.996460 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.997389 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.997854 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.997987 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.000246 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:38.000252 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:38.000255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:38.000290 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:38.002313 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.002930 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.002934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.002937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.003046 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.003431 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.003434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.003436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.003722 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:38.003819 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:38.004296 (+ 477us) server_negotiation.cc:299] Negotiation successful
0810 06:20:38.004421 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:20:38.004628 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.975977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.194:37693)
0810 06:20:37.976536 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.976586 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.977534 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.978085 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.978094 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:37.995586 (+ 17492us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:37.995694 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.995701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.996601 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.996603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.997252 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:37.997268 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:37.997972 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:37.997975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:37.998065 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:37.998565 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:37.998576 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:38.000086 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:38.002465 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:38.002470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:38.002473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:38.002788 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:38.003181 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:38.003184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:38.003187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:38.003297 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:38.003907 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:38.003910 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:38.004036 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 06:20:38.004194 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 06:20:38.004371 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"spinlock_wait_cycles":35840,"thread_start_us":252,"threads_started":1}
I20260810 06:20:38.005777 12101 heartbeater.cc:347] Connected to a master server at 127.31.204.254:34003
I20260810 06:20:38.006109 12101 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:38.006680 12101 heartbeater.cc:511] Master 127.31.204.254:34003 requested a full tablet report, sending...
I20260810 06:20:38.008170 12102 heartbeater.cc:347] Connected to a master server at 127.31.204.253:36451
I20260810 06:20:38.008260 12102 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:38.008415 12102 heartbeater.cc:511] Master 127.31.204.253:36451 requested a full tablet report, sending...
I20260810 06:20:38.009125 11273 ts_manager.cc:194] Registered new tserver with Master: bc19037c918f422ba77c253d7c5eec17 (127.31.204.194:46103)
I20260810 06:20:38.009204 10241 ts_manager.cc:194] Re-registered known tserver with Master: bc19037c918f422ba77c253d7c5eec17 (127.31.204.194:46103)
I20260810 06:20:38.009886 10241 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:37693
Aug 10 06:20:38 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342837, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.252@KRBTEST.COM
I20260810 06:20:38.013095 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:43569
--local_ip_for_outbound_sockets=127.31.204.195
--tserver_master_addrs=127.31.204.254:34003
--webserver_port=41459
--webserver_interface=127.31.204.195
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.195.keytab
--principal=oryx/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.204.254:34003,127.31.204.253:36451
--tserver_master_addrs=127.31.204.254:34003,127.31.204.253:36451,127.31.204.252:40965 with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:20:38.015376 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.975977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:40965 (local address 127.31.204.194:42255)
0810 06:20:37.976568 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:37.976590 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:37.977541 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:37.978086 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:37.978094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:38.006396 (+ 28302us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:38.006511 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.006519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.007588 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.007591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.008387 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.008398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.010127 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.010131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.010278 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.010727 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:38.010741 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:38.012014 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:38.013936 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:38.013940 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:38.013941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:38.014192 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:38.014445 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:38.014448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:38.014449 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:38.014539 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:38.014994 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:38.014996 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:38.015067 (+ 71us) client_negotiation.cc:769] Sending connection context
0810 06:20:38.015138 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 06:20:38.015246 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":40,"spinlock_wait_cycles":3840,"thread_start_us":146,"threads_started":1}
I20260810 06:20:38.015506 11886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:37.975565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:42255 (local address 127.31.204.252:40965)
0810 06:20:37.975714 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:37.975718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:37.976677 (+ 959us) server_negotiation.cc:407] Connection header received
0810 06:20:37.977735 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:37.977739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:37.977802 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:37.977874 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:38.006669 (+ 28795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.007445 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.008555 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.008908 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.009008 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.012156 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:38.012159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:38.012161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:38.012187 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:38.013821 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.014284 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.014287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.014288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.014361 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.014660 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.014664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.014665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.014904 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:38.014998 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:38.015222 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 06:20:38.015351 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 06:20:38.016438 12098 heartbeater.cc:347] Connected to a master server at 127.31.204.252:40965
I20260810 06:20:38.016515 12098 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:38.016669 12098 heartbeater.cc:511] Master 127.31.204.252:40965 requested a full tablet report, sending...
I20260810 06:20:38.017343 11590 ts_manager.cc:194] Registered new tserver with Master: bc19037c918f422ba77c253d7c5eec17 (127.31.204.194:46103)
W20260810 06:20:38.123433 12110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:38.123667 12110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:38.123723 12110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:38.127792 12110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:38.127844 12110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:38.127919 12110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:20:38.133077 12110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.195.keytab
--principal=oryx/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:43569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=41459
--tserver_master_addrs=127.31.204.254:34003,127.31.204.253:36451,127.31.204.252:40965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:38.133996 12110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:38.135001 12110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:20:38.136037 11886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.132663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36064 (local address 127.31.204.252:40965)
0810 06:20:38.132791 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:38.132795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:38.132914 (+ 119us) server_negotiation.cc:407] Connection header received
0810 06:20:38.133056 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:38.133058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:38.133112 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:38.133189 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:38.133447 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.133947 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.135043 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.135509 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.135617 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.135779 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:38.135842 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 06:20:38.135912 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:20:38.136051 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.132592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:40965 (local address 127.0.0.1:36064)
0810 06:20:38.132834 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:38.132846 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:38.132957 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:38.133215 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:38.133218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:38.133237 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:38.133344 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.133350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.134074 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.134078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.134912 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.134922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.135624 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.135628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.135718 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.135742 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 06:20:38.135828 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 06:20:38.135927 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":68,"threads_started":1}
I20260810 06:20:38.136833 11608 raft_consensus.cc:1203] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 06:20:38.137281 11656 sys_catalog.cc:455] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf988718c8da472c975d2fa2b1e2a341. Latest consensus state: current_term: 1 leader_uuid: "cf988718c8da472c975d2fa2b1e2a341" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: NON_VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: true } } }
I20260810 06:20:38.137421 11656 sys_catalog.cc:458] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [sys.catalog]: This master's current role is: LEARNER
W20260810 06:20:38.144464 12121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:38.144467 12122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:38.144577 12124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:38.144774 12110 server_base.cc:1034] running on GCE node
I20260810 06:20:38.145146 12110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:38.145705 12110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:38.146858 12110 hybrid_clock.cc:648] HybridClock initialized: now 1786342838146841 us; error 28 us; skew 500 ppm
Aug 10 06:20:38 dist-test-slave-q9h9 krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342838, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:38.149431 12110 init.cc:377] Logged in from keytab as oryx/127.31.204.195@KRBTEST.COM (short username oryx)
I20260810 06:20:38.150285 12110 webserver.cc:460] Webserver started at http://127.31.204.195:41459/ using document root <none> and password file <none>
I20260810 06:20:38.150795 12110 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:38.150872 12110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:38.154595 12110 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:38.156642 12136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:38.157841 12110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:38.157949 12110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "b59b0c145b014a6ea2444c33b301971c"
format_stamp: "Formatted at 2026-08-10 06:20:36 on dist-test-slave-q9h9"
server_key: "3b54804321c425d3e645acb0154f63bd"
server_key_iv: "a699813affcfd1e9978a303ee1ad66e5"
server_key_version: "encryptionkey@0"
I20260810 06:20:38.158494 12110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:38.187085 12110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:38.190220 12110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:38.190403 12110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:38.190980 12110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:38.191840 12110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:38.191890 12110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:38.191942 12110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:38.191972 12110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:38.201930 12110 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:43569
I20260810 06:20:38.201956 12314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:43569 every 8 connection(s)
I20260810 06:20:38.202898 12110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:20:38.209870 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 12110
Aug 10 06:20:38 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342838, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.253@KRBTEST.COM
Aug 10 06:20:38 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342838, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM
Aug 10 06:20:38 dist-test-slave-q9h9 krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342838, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.252@KRBTEST.COM
I20260810 06:20:38.218954 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.205319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:35577 (local address 127.31.204.253:36451)
0810 06:20:38.205438 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:38.205442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:38.205970 (+ 528us) server_negotiation.cc:407] Connection header received
0810 06:20:38.206914 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:38.206918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:38.206972 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:38.207023 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:38.209417 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.210220 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.211269 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.211597 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.211671 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.214750 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:38.214754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:38.214757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:38.214785 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:38.216677 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.217268 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.217271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.217273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.217367 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.217746 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.217749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.217751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.217958 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:38.218054 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:38.218676 (+ 622us) server_negotiation.cc:299] Negotiation successful
0810 06:20:38.218798 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:20:38.219643 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.205657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:35111 (local address 127.31.204.254:34003)
0810 06:20:38.205771 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:38.205776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:38.205846 (+ 70us) server_negotiation.cc:407] Connection header received
0810 06:20:38.206914 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:38.206918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:38.206972 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:38.207023 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:38.208449 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.209272 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.210966 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.211279 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.211361 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.215482 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:38.215487 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:38.215489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:38.215516 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:38.217406 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.218099 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.218103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.218104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.218195 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.218757 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.218761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.218763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.218963 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:38.219046 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:38.219375 (+ 329us) server_negotiation.cc:299] Negotiation successful
0810 06:20:38.219483 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 06:20:38.220427 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.205134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:36451 (local address 127.31.204.195:35577)
0810 06:20:38.205895 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:38.205907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:38.206670 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:38.207329 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:38.207336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:38.207747 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:38.208275 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.208291 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.210346 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.210349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.211152 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.211160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.211817 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.211820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.211912 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.212451 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:38.212471 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:38.214568 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:38.216814 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:38.216818 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:38.216832 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:38.217157 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:38.217474 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:38.217477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:38.217479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:38.217643 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:38.218147 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:38.218153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:38.218412 (+ 259us) client_negotiation.cc:769] Sending connection context
0810 06:20:38.219152 (+ 740us) client_negotiation.cc:240] Negotiation successful
0810 06:20:38.219444 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":160,"spinlock_wait_cycles":48512,"thread_start_us":144,"threads_started":1}
I20260810 06:20:38.220750 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.205133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.195:35111)
0810 06:20:38.205699 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:38.205753 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:38.206680 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:38.207329 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:38.207336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:38.207748 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:38.208275 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.208291 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.209425 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.209428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.210806 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.210818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.211560 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.211563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.211742 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.212516 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:38.212530 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:38.215325 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:38.217514 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:38.217517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:38.217519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:38.217745 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:38.218314 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:38.218317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:38.218319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:38.218639 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:38.219104 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:38.219107 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:38.219194 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 06:20:38.219784 (+ 590us) client_negotiation.cc:240] Negotiation successful
0810 06:20:38.219954 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":95,"spinlock_wait_cycles":11520,"thread_start_us":254,"threads_started":1}
I20260810 06:20:38.221537 12316 heartbeater.cc:347] Connected to a master server at 127.31.204.254:34003
I20260810 06:20:38.221769 12316 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:38.222064 12318 heartbeater.cc:347] Connected to a master server at 127.31.204.253:36451
I20260810 06:20:38.222144 12318 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:38.222363 12316 heartbeater.cc:511] Master 127.31.204.254:34003 requested a full tablet report, sending...
I20260810 06:20:38.222363 12318 heartbeater.cc:511] Master 127.31.204.253:36451 requested a full tablet report, sending...
I20260810 06:20:38.222962 11886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.205318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:46091 (local address 127.31.204.252:40965)
0810 06:20:38.205438 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:38.205442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:38.206474 (+ 1032us) server_negotiation.cc:407] Connection header received
0810 06:20:38.206910 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:38.206913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:38.206969 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:38.207014 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:38.208431 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.209287 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.211041 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.211387 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.211458 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.217619 (+ 6161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:38.217622 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:38.217625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:38.217651 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:38.219186 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.220763 (+ 1577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.220768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.220769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.220863 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.221947 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.221951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.221953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.222298 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:38.222374 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:38.222704 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 06:20:38.222825 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:20:38.223439 12322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.205134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:40965 (local address 127.31.204.195:46091)
0810 06:20:38.206372 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:38.206383 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:38.206671 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:38.207328 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:38.207335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:38.207745 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:38.208274 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.208287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.209430 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.209434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.210894 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.210904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.211616 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.211620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.211743 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.213489 (+ 1746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:38.213504 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:38.217479 (+ 3975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:38.219579 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:38.219582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:38.219584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:38.220021 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:38.221309 (+ 1288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:38.221311 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:38.221313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:38.221492 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:38.222431 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:38.222433 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:38.222516 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 06:20:38.223145 (+ 629us) client_negotiation.cc:240] Negotiation successful
0810 06:20:38.223279 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1166,"mutex_wait_us":975,"thread_start_us":91,"threads_started":1}
I20260810 06:20:38.223485 10241 ts_manager.cc:194] Re-registered known tserver with Master: b59b0c145b014a6ea2444c33b301971c (127.31.204.195:43569)
I20260810 06:20:38.223692 11273 ts_manager.cc:194] Registered new tserver with Master: b59b0c145b014a6ea2444c33b301971c (127.31.204.195:43569)
I20260810 06:20:38.223984 12315 heartbeater.cc:347] Connected to a master server at 127.31.204.252:40965
I20260810 06:20:38.224061 12315 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:38.224222 12315 heartbeater.cc:511] Master 127.31.204.252:40965 requested a full tablet report, sending...
I20260810 06:20:38.224223 10241 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:35111
I20260810 06:20:38.224876 11590 ts_manager.cc:194] Registered new tserver with Master: b59b0c145b014a6ea2444c33b301971c (127.31.204.195:43569)
I20260810 06:20:38.227886 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.212059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36080 (local address 127.31.204.252:40965)
0810 06:20:38.212510 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:38.212514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:38.214282 (+ 1768us) server_negotiation.cc:407] Connection header received
0810 06:20:38.214327 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:38.214331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:38.214378 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:38.214452 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:38.215324 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.216120 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.220234 (+ 4114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.220556 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.220667 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.223299 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:38.223302 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:38.223304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:38.223329 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:38.226235 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.226690 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.226694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.226695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.226784 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.227047 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.227050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.227052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.227270 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:38.227395 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:38.227640 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 06:20:38.227747 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":93,"threads_started":1}
I20260810 06:20:38.228349 12337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.212847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35704 (local address 127.31.204.253:36451)
0810 06:20:38.213367 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:38.213371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:38.214006 (+ 635us) server_negotiation.cc:407] Connection header received
0810 06:20:38.214071 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:38.214074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:38.214121 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:38.214201 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:38.217026 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.217715 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.219650 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.220037 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.220131 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.224690 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:38.224694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:38.224744 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:38.224771 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:38.226684 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.227090 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.227093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.227095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.227169 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.227466 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.227469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.227471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.227691 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:38.227804 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:38.228083 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 06:20:38.228200 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":75,"threads_started":1}
I20260810 06:20:38.229177 12330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.212050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50946 (local address 127.31.204.254:34003)
0810 06:20:38.212423 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:38.212427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:38.213189 (+ 762us) server_negotiation.cc:407] Connection header received
0810 06:20:38.213230 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:38.213233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:38.213282 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:38.213358 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:38.214350 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.215149 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.216305 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.216638 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.216734 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.225447 (+ 8713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:38.225451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:38.225454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:38.225479 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:38.227116 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.227476 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.227479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.227481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.227560 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.228134 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.228138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.228139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.228339 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:38.228659 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:38.228902 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 06:20:38.229027 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":93,"threads_started":1}
I20260810 06:20:38.231514 10241 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:20:38.234344 10241 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:20:38.236864 12345 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cf988718c8da472c975d2fa2b1e2a341" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: NON_VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: true } } }
I20260810 06:20:38.236992 12345 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:38.239995 12345 raft_consensus.cc:1050] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341: attempting to promote NON_VOTER db14022bd7e64ec4873973f0166a1fc8 to VOTER
I20260810 06:20:38.240293 12345 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } }
I20260810 06:20:38.241377 11608 raft_consensus.cc:1261] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 1 LEARNER]: Refusing update from remote peer cf988718c8da472c975d2fa2b1e2a341: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 06:20:38.241672 11289 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 1 FOLLOWER]: Refusing update from remote peer cf988718c8da472c975d2fa2b1e2a341: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 06:20:38.241721 12342 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 06:20:38.241995 12342 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 06:20:38.243407 12111 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, db14022bd7e64ec4873973f0166a1fc8 (127.31.204.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } } }
I20260810 06:20:38.244830 12111 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cf988718c8da472c975d2fa2b1e2a341" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } } }
I20260810 06:20:38.244931 12111 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:38.244894 11289 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, db14022bd7e64ec4873973f0166a1fc8 (127.31.204.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } } }
I20260810 06:20:38.244975 12345 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cf988718c8da472c975d2fa2b1e2a341" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } } }
I20260810 06:20:38.245060 12345 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:38.245453 12111 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cf988718c8da472c975d2fa2b1e2a341" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } } }
I20260810 06:20:38.245551 12111 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341 [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:38.246454 12355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cf988718c8da472c975d2fa2b1e2a341" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } } }
I20260810 06:20:38.246646 12355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:20:38.246419 11608 raft_consensus.cc:2932] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, db14022bd7e64ec4873973f0166a1fc8 (127.31.204.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } } }
I20260810 06:20:38.248073 11656 sys_catalog.cc:455] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cf988718c8da472c975d2fa2b1e2a341" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } } }
I20260810 06:20:38.248179 11656 sys_catalog.cc:458] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:20:38.261680 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.255915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:46103 (local address 127.0.0.1:49324)
0810 06:20:38.256066 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:38.256080 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:38.256181 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:38.257020 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:38.257023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:38.257040 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:38.257138 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.257143 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.259255 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.259258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.260310 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.260321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.261288 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.261292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.261371 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.261391 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 06:20:38.261469 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 06:20:38.261543 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 06:20:38.261736 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.256073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.31.204.194:46103)
0810 06:20:38.256509 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:38.256527 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:38.256550 (+ 23us) server_negotiation.cc:407] Connection header received
0810 06:20:38.256642 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:38.256648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:38.256862 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:38.257287 (+ 425us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:38.257433 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.258532 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.260437 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.261170 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.261331 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.261471 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:38.261520 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 06:20:38.261599 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":108,"threads_started":1}
I20260810 06:20:38.262269 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.256401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:41705 (local address 127.0.0.1:55820)
0810 06:20:38.257041 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:38.257053 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:38.257133 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:38.258310 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:38.258314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:38.258332 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:38.258426 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.258432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.259912 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.259916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.260824 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.260832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.261861 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.261865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.261951 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.261974 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 06:20:38.262048 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 06:20:38.262141 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":163,"threads_started":1}
I20260810 06:20:38.262547 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.256763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:43569 (local address 127.0.0.1:42078)
0810 06:20:38.257507 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:38.257520 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:38.257609 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:38.258214 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:38.258217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:38.258240 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:38.258319 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.258324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.259729 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.259732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.260796 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.260807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.262109 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.262113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.262198 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.262220 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:20:38.262309 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 06:20:38.262425 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":62,"thread_start_us":123,"threads_started":1}
I20260810 06:20:38.262885 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.257172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.31.204.195:43569)
0810 06:20:38.257492 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:38.257504 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:38.257588 (+ 84us) server_negotiation.cc:407] Connection header received
0810 06:20:38.257711 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:38.257717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:38.257902 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:38.258069 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:38.258432 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.259590 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.260934 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.261717 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.262111 (+ 394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.262230 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:38.262496 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 06:20:38.262748 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":99,"threads_started":1}
I20260810 06:20:38.262949 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.256572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55820 (local address 127.31.204.193:41705)
0810 06:20:38.257825 (+ 1253us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:38.257840 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:38.257861 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:20:38.257932 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:38.257937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:38.258154 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:38.258556 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:38.258679 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.259791 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.260950 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.261717 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.262550 (+ 833us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.262683 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:38.262727 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 06:20:38.262816 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1156,"thread_start_us":71,"threads_started":1}
I20260810 06:20:38.264907 11992 tablet_service.cc:1467] Processing CreateTablet for tablet effa5afc60254d249ac926bdc59adbea (DEFAULT_TABLE table=test-table [id=c22e0b8e78904b2cb2ec18e24d64c742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:38.265311 11776 tablet_service.cc:1467] Processing CreateTablet for tablet effa5afc60254d249ac926bdc59adbea (DEFAULT_TABLE table=test-table [id=c22e0b8e78904b2cb2ec18e24d64c742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:38.266055 11992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet effa5afc60254d249ac926bdc59adbea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:38.265715 12207 tablet_service.cc:1467] Processing CreateTablet for tablet effa5afc60254d249ac926bdc59adbea (DEFAULT_TABLE table=test-table [id=c22e0b8e78904b2cb2ec18e24d64c742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:38.266232 11776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet effa5afc60254d249ac926bdc59adbea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:38.266644 12207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet effa5afc60254d249ac926bdc59adbea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:38.273641 12367 tablet_bootstrap.cc:492] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237: Bootstrap starting.
I20260810 06:20:38.274770 12369 tablet_bootstrap.cc:492] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c: Bootstrap starting.
I20260810 06:20:38.274744 12368 tablet_bootstrap.cc:492] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17: Bootstrap starting.
I20260810 06:20:38.275992 12367 tablet_bootstrap.cc:654] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:38.276901 12368 tablet_bootstrap.cc:654] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:38.276901 12369 tablet_bootstrap.cc:654] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:38.277248 12367 log.cc:824] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:38.277740 12369 log.cc:824] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:38.277747 12368 log.cc:824] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:38.279263 12367 tablet_bootstrap.cc:492] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237: No bootstrap required, opened a new log
I20260810 06:20:38.279454 12367 ts_tablet_manager.cc:1397] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:20:38.279731 12368 tablet_bootstrap.cc:492] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17: No bootstrap required, opened a new log
I20260810 06:20:38.279922 12368 ts_tablet_manager.cc:1397] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260810 06:20:38.280010 12369 tablet_bootstrap.cc:492] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c: No bootstrap required, opened a new log
I20260810 06:20:38.280196 12369 ts_tablet_manager.cc:1397] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 06:20:38.282405 12367 raft_consensus.cc:348] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } } peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } }
I20260810 06:20:38.282614 12367 raft_consensus.cc:374] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:38.282667 12367 raft_consensus.cc:729] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e764aab3b1344241ac6bd8bc5639a237, State: Initialized, Role: FOLLOWER
I20260810 06:20:38.283082 12367 consensus_queue.cc:260] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } } peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } }
I20260810 06:20:38.283721 11876 heartbeater.cc:503] Master 127.31.204.254:34003 was elected leader, sending a full tablet report...
I20260810 06:20:38.284399 12368 raft_consensus.cc:348] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } } peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } }
I20260810 06:20:38.284650 12368 raft_consensus.cc:374] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:38.284703 12368 raft_consensus.cc:729] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc19037c918f422ba77c253d7c5eec17, State: Initialized, Role: FOLLOWER
I20260810 06:20:38.285095 12368 consensus_queue.cc:260] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } } peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } }
I20260810 06:20:38.286101 12368 ts_tablet_manager.cc:1428] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260810 06:20:38.286113 12367 ts_tablet_manager.cc:1428] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260810 06:20:38.286456 12101 heartbeater.cc:503] Master 127.31.204.254:34003 was elected leader, sending a full tablet report...
I20260810 06:20:38.289301 12369 raft_consensus.cc:348] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } } peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } }
I20260810 06:20:38.289547 12369 raft_consensus.cc:374] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:38.289608 12369 raft_consensus.cc:729] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b59b0c145b014a6ea2444c33b301971c, State: Initialized, Role: FOLLOWER
I20260810 06:20:38.290036 12369 consensus_queue.cc:260] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } } peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } }
I20260810 06:20:38.290735 12316 heartbeater.cc:503] Master 127.31.204.254:34003 was elected leader, sending a full tablet report...
I20260810 06:20:38.290890 12369 ts_tablet_manager.cc:1428] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260810 06:20:38.312372 12378 raft_consensus.cc:482] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:20:38.312557 12378 raft_consensus.cc:504] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } } peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } }
I20260810 06:20:38.313651 12378 leader_election.cc:288] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc19037c918f422ba77c253d7c5eec17 (127.31.204.194:46103), b59b0c145b014a6ea2444c33b301971c (127.31.204.195:43569)
I20260810 06:20:38.318564 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.313928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:59449 (local address 127.31.204.194:46103)
0810 06:20:38.314074 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:38.314078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:38.314320 (+ 242us) server_negotiation.cc:407] Connection header received
0810 06:20:38.314417 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:38.314420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:38.314467 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:38.314550 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:38.314916 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.315710 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.317339 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.317964 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.318109 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.318157 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:38.318391 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 06:20:38.318459 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 06:20:38.318574 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.313850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:46103 (local address 127.31.204.193:59449)
0810 06:20:38.314241 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:38.314251 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:38.314321 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:38.314577 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:38.314581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:38.314612 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:38.314793 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.314799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.315819 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.315822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.317187 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.317199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.318123 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.318127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.318231 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.318286 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 06:20:38.318351 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 06:20:38.318431 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":105,"spinlock_wait_cycles":6784,"thread_start_us":63,"threads_started":1}
I20260810 06:20:38.318710 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.313949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:37771 (local address 127.31.204.195:43569)
0810 06:20:38.314075 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:38.314079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:38.314283 (+ 204us) server_negotiation.cc:407] Connection header received
0810 06:20:38.314417 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:38.314420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:38.314467 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:38.314550 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:38.314916 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.315706 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.317339 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.317940 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.318103 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.318159 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:38.318508 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 06:20:38.318579 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:20:38.319113 12020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "effa5afc60254d249ac926bdc59adbea" candidate_uuid: "e764aab3b1344241ac6bd8bc5639a237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc19037c918f422ba77c253d7c5eec17" is_pre_election: true
I20260810 06:20:38.319185 12391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.313852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:43569 (local address 127.31.204.193:37771)
0810 06:20:38.314193 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:38.314209 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:38.314320 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:38.314577 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:38.314581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:38.314614 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:38.314786 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.314793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.315819 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.315822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.317189 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:38.317199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.318244 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.318248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:38.318372 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.318398 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 06:20:38.318468 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 06:20:38.318547 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":151,"spinlock_wait_cycles":8064,"thread_start_us":85,"threads_started":1}
I20260810 06:20:38.319391 12020 raft_consensus.cc:2454] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e764aab3b1344241ac6bd8bc5639a237 in term 0.
I20260810 06:20:38.319757 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "effa5afc60254d249ac926bdc59adbea" candidate_uuid: "e764aab3b1344241ac6bd8bc5639a237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b59b0c145b014a6ea2444c33b301971c" is_pre_election: true
I20260810 06:20:38.319837 11702 leader_election.cc:302] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc19037c918f422ba77c253d7c5eec17, e764aab3b1344241ac6bd8bc5639a237; no voters:
I20260810 06:20:38.319995 12248 raft_consensus.cc:2454] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e764aab3b1344241ac6bd8bc5639a237 in term 0.
I20260810 06:20:38.320102 12378 raft_consensus.cc:2781] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:20:38.320190 12378 raft_consensus.cc:482] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:20:38.320245 12378 raft_consensus.cc:3037] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:38.321238 12378 raft_consensus.cc:504] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } } peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } }
I20260810 06:20:38.321569 12378 leader_election.cc:288] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [CANDIDATE]: Term 1 election: Requested vote from peers bc19037c918f422ba77c253d7c5eec17 (127.31.204.194:46103), b59b0c145b014a6ea2444c33b301971c (127.31.204.195:43569)
I20260810 06:20:38.321892 12020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "effa5afc60254d249ac926bdc59adbea" candidate_uuid: "e764aab3b1344241ac6bd8bc5639a237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc19037c918f422ba77c253d7c5eec17"
I20260810 06:20:38.321913 12248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "effa5afc60254d249ac926bdc59adbea" candidate_uuid: "e764aab3b1344241ac6bd8bc5639a237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b59b0c145b014a6ea2444c33b301971c"
I20260810 06:20:38.322000 12248 raft_consensus.cc:3037] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:38.322006 12020 raft_consensus.cc:3037] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:38.323006 12020 raft_consensus.cc:2454] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e764aab3b1344241ac6bd8bc5639a237 in term 1.
I20260810 06:20:38.323199 12248 raft_consensus.cc:2454] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e764aab3b1344241ac6bd8bc5639a237 in term 1.
I20260810 06:20:38.323271 11702 leader_election.cc:302] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc19037c918f422ba77c253d7c5eec17, e764aab3b1344241ac6bd8bc5639a237; no voters:
I20260810 06:20:38.323434 12378 raft_consensus.cc:2781] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:38.323606 12378 raft_consensus.cc:686] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [term 1 LEADER]: Becoming Leader. State: Replica: e764aab3b1344241ac6bd8bc5639a237, State: Running, Role: LEADER
I20260810 06:20:38.323952 12378 consensus_queue.cc:237] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } } peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } }
I20260810 06:20:38.326802 10241 catalog_manager.cc:5495] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 reported cstate change: term changed from 0 to 1, leader changed from <none> to e764aab3b1344241ac6bd8bc5639a237 (127.31.204.193). New cstate: current_term: 1 leader_uuid: "e764aab3b1344241ac6bd8bc5639a237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } health_report { overall_health: UNKNOWN } } }
I20260810 06:20:38.350556 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.347276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55828 (local address 127.31.204.193:41705)
0810 06:20:38.347396 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:38.347399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:38.347412 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:20:38.347478 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:38.347480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:38.347547 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:38.347636 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:38.347823 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.348356 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.349223 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.349442 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.349531 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.349791 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:38.350225 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:38.350317 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:38.350367 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 06:20:38.350416 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:20:38.355919 12248 raft_consensus.cc:1261] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c [term 1 FOLLOWER]: Refusing update from remote peer e764aab3b1344241ac6bd8bc5639a237: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:38.355921 12020 raft_consensus.cc:1261] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17 [term 1 FOLLOWER]: Refusing update from remote peer e764aab3b1344241ac6bd8bc5639a237: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:38.356606 12393 consensus_queue.cc:1035] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:38.356775 12378 consensus_queue.cc:1035] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:38.364626 12403 mvcc.cc:205] Tried to move back new op lower bound from 7316860265899495424 to 7316860265777381376. Current Snapshot: MvccSnapshot[applied={T|T < 7316860265899495424}]
I20260810 06:20:38.374850 10240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:38.375018 10240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:38.380102 10240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf988718c8da472c975d2fa2b1e2a341: Sending DeleteTablet for 3 replicas of tablet effa5afc60254d249ac926bdc59adbea
I20260810 06:20:38.380832 12207 tablet_service.cc:1514] Processing DeleteTablet for tablet effa5afc60254d249ac926bdc59adbea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:38 UTC) from {username='oryx', principal='oryx/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:42078
I20260810 06:20:38.380841 11992 tablet_service.cc:1514] Processing DeleteTablet for tablet effa5afc60254d249ac926bdc59adbea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:38 UTC) from {username='oryx', principal='oryx/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:49324
I20260810 06:20:38.381290 12416 tablet_replica.cc:330] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c: stopping tablet replica
I20260810 06:20:38.382198 11776 tablet_service.cc:1514] Processing DeleteTablet for tablet effa5afc60254d249ac926bdc59adbea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:38 UTC) from {username='oryx', principal='oryx/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:55820
I20260810 06:20:38.382265 12416 raft_consensus.cc:2229] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:38.382558 12416 raft_consensus.cc:2258] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:38.382663 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 10195
I20260810 06:20:38.390512 12423 tablet_replica.cc:330] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237: stopping tablet replica
I20260810 06:20:38.390799 12423 raft_consensus.cc:2229] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:20:38.391139 12423 raft_consensus.cc:2258] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:38.391673 12416 ts_tablet_manager.cc:1905] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:38.392369 12423 ts_tablet_manager.cc:1905] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:38.394433 12416 ts_tablet_manager.cc:1918] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:20:38.394532 12416 log.cc:1196] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/effa5afc60254d249ac926bdc59adbea
I20260810 06:20:38.394912 12416 ts_tablet_manager.cc:1939] T effa5afc60254d249ac926bdc59adbea P b59b0c145b014a6ea2444c33b301971c: Deleting consensus metadata
I20260810 06:20:38.395390 12423 ts_tablet_manager.cc:1918] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:20:38.395488 12423 log.cc:1196] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-0/wal/wals/effa5afc60254d249ac926bdc59adbea
I20260810 06:20:38.395805 12423 ts_tablet_manager.cc:1939] T effa5afc60254d249ac926bdc59adbea P e764aab3b1344241ac6bd8bc5639a237: Deleting consensus metadata
I20260810 06:20:38.396247 12415 tablet_replica.cc:330] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17: stopping tablet replica
I20260810 06:20:38.396552 12415 raft_consensus.cc:2229] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:38.396874 12415 raft_consensus.cc:2258] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:38.398042 12415 ts_tablet_manager.cc:1905] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 06:20:38.399822 12138 connection.cc:535] server connection from 127.0.0.1:42078 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 06:20:38.400223 11701 connection.cc:535] server connection from 127.0.0.1:55820 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 06:20:38.400632 12415 ts_tablet_manager.cc:1918] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:20:38.400741 12415 log.cc:1196] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/effa5afc60254d249ac926bdc59adbea
I20260810 06:20:38.401059 12415 ts_tablet_manager.cc:1939] T effa5afc60254d249ac926bdc59adbea P bc19037c918f422ba77c253d7c5eec17: Deleting consensus metadata
W20260810 06:20:38.402114 11916 connection.cc:441] server connection from 127.0.0.1:49324 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49324 (request call id 1) could send its response
I20260810 06:20:38.404166 12337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.382822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35706 (local address 127.31.204.253:36451)
0810 06:20:38.385253 (+ 2431us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:38.385257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:38.385272 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:20:38.385312 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:38.385315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:38.385365 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:38.385452 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:38.391760 (+ 6308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.392568 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.398163 (+ 5595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.398500 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.398579 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.400509 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:38.400513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:38.400516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:38.400544 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:38.402220 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.402720 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.402724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.402726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.402820 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.403147 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.403163 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.403165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.403401 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:38.403514 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:38.403872 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 06:20:38.404009 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2344}
I20260810 06:20:38.414172 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.388255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36086 (local address 127.31.204.252:40965)
0810 06:20:38.389162 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:38.389167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:38.404242 (+ 15075us) server_negotiation.cc:407] Connection header received
0810 06:20:38.404403 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:38.404407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:38.404467 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:38.404540 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:38.405382 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.406121 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.407189 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.408366 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.408464 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.410188 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:38.410192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:38.410195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:38.410224 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:38.412058 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.412529 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.412532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.412535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.412625 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.412938 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.412942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.412944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.413138 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:38.413263 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:38.413821 (+ 558us) server_negotiation.cc:299] Negotiation successful
0810 06:20:38.413931 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":803,"server-negotiator.queue_time_us":833}
I20260810 06:20:38.414307 12337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.405416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35722 (local address 127.31.204.253:36451)
0810 06:20:38.405516 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:38.405520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:38.405532 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:20:38.405563 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:38.405566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:38.405604 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:38.405702 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:38.406950 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.407640 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.408421 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.408736 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.408831 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.410335 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:38.410338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:38.410340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:38.410363 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:38.412475 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.412949 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.412953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.412954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.413034 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.413324 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.413327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.413329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.413533 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:38.413635 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:38.413973 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 06:20:38.414163 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 06:20:38.414990 11886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:38.405083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36100 (local address 127.31.204.252:40965)
0810 06:20:38.405204 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:38.405209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:38.406199 (+ 990us) server_negotiation.cc:407] Connection header received
0810 06:20:38.406363 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:38.406366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:38.406409 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:38.406455 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:38.407365 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.408153 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.409610 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:38.409950 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:38.410050 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:38.411642 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:38.411646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:38.411649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:38.411688 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:38.413152 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.413638 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.413641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.413643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.413737 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:38.414097 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:38.414101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:38.414102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:38.414290 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:38.414383 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:38.414728 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 06:20:38.414855 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":44,"spinlock_wait_cycles":1280}
I20260810 06:20:39.367846 12433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:39.367425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.195:59693)
0810 06:20:39.367645 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:39.367753 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":71,"threads_started":1}
W20260810 06:20:39.368139 12316 heartbeater.cc:644] Failed to heartbeat to 127.31.204.254:34003 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
I20260810 06:20:39.369688 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:39.369272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.194:53275)
0810 06:20:39.369517 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:39.369599 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":61,"threads_started":1}
I20260810 06:20:39.370371 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:39.369971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.193:49773)
0810 06:20:39.370184 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:39.370275 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
W20260810 06:20:39.370401 12101 heartbeater.cc:644] Failed to heartbeat to 127.31.204.254:34003 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
W20260810 06:20:39.370713 11876 heartbeater.cc:644] Failed to heartbeat to 127.31.204.254:34003 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
I20260810 06:20:39.879974 12436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cf988718c8da472c975d2fa2b1e2a341)
I20260810 06:20:39.880167 12436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } }
I20260810 06:20:39.880609 12437 raft_consensus.cc:482] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cf988718c8da472c975d2fa2b1e2a341)
I20260810 06:20:39.880776 12437 raft_consensus.cc:504] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } }
I20260810 06:20:39.881222 12436 leader_election.cc:288] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf988718c8da472c975d2fa2b1e2a341 (127.31.204.254:34003), db14022bd7e64ec4873973f0166a1fc8 (127.31.204.252:40965)
I20260810 06:20:39.881731 12437 leader_election.cc:288] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf988718c8da472c975d2fa2b1e2a341 (127.31.204.254:34003), 786f5f83aad243f5a9f6313c2c3681da (127.31.204.253:36451)
I20260810 06:20:39.882089 12440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:39.881357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:51336)
0810 06:20:39.881774 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:39.881965 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":71,"threads_started":1}
W20260810 06:20:39.882370 11251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
I20260810 06:20:39.882562 12445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:39.881814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:51348)
0810 06:20:39.882394 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:39.882472 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":68,"threads_started":1}
W20260810 06:20:39.882709 11566 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
I20260810 06:20:39.883464 12440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:39.883232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:51356)
0810 06:20:39.883358 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:39.883403 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 06:20:39.883607 12445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:39.883377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:51370)
0810 06:20:39.883504 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:39.883547 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260810 06:20:39.883639 11251 leader_election.cc:334] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cf988718c8da472c975d2fa2b1e2a341 (127.31.204.254:34003): Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
W20260810 06:20:39.883729 11566 leader_election.cc:334] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cf988718c8da472c975d2fa2b1e2a341 (127.31.204.254:34003): Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
I20260810 06:20:39.886164 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:39.881450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:40965 (local address 127.0.0.1:36110)
0810 06:20:39.882096 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:39.882112 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:39.882212 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:39.882481 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:39.882484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:39.882504 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:39.882620 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:39.882627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:39.883708 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:39.883711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:39.884998 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:39.885009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:39.885734 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:39.885737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:39.885820 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:39.885867 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 06:20:39.885929 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 06:20:39.886001 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":45,"thread_start_us":85,"threads_started":1}
I20260810 06:20:39.886164 12442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:39.881642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36110 (local address 127.31.204.252:40965)
0810 06:20:39.882231 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:39.882235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:39.882254 (+ 19us) server_negotiation.cc:407] Connection header received
0810 06:20:39.882303 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:39.882306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:39.882353 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:39.882732 (+ 379us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:39.882835 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:39.883580 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:39.885137 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:39.885609 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:39.885710 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:39.885761 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:39.885963 (+ 202us) server_negotiation.cc:299] Negotiation successful
0810 06:20:39.886056 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":79,"threads_started":1}
I20260810 06:20:39.886294 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:39.881926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:36451 (local address 127.0.0.1:35734)
0810 06:20:39.882467 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:39.882479 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:39.882567 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:39.882939 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:39.882943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:39.882961 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:39.883050 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:39.883057 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:39.884059 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:39.884062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:39.885233 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:39.885241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:39.885872 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:39.885876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:39.885966 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:39.885984 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 06:20:39.886085 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 06:20:39.886162 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":96,"threads_started":1}
I20260810 06:20:39.886307 12447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:39.882033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35734 (local address 127.31.204.253:36451)
0810 06:20:39.882273 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:39.882277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:39.882649 (+ 372us) server_negotiation.cc:407] Connection header received
0810 06:20:39.882708 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:39.882711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:39.882754 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:39.882827 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:39.883178 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:39.883932 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:39.885345 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:39.885747 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:39.885872 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:39.885920 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:39.886104 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 06:20:39.886178 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":181,"thread_start_us":59,"threads_started":1}
I20260810 06:20:39.886929 11289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db14022bd7e64ec4873973f0166a1fc8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "786f5f83aad243f5a9f6313c2c3681da" is_pre_election: true
I20260810 06:20:39.886929 11608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "786f5f83aad243f5a9f6313c2c3681da" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "db14022bd7e64ec4873973f0166a1fc8" is_pre_election: true
I20260810 06:20:39.887118 11289 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db14022bd7e64ec4873973f0166a1fc8 in term 1.
I20260810 06:20:39.887118 11608 raft_consensus.cc:2454] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f5f83aad243f5a9f6313c2c3681da in term 1.
I20260810 06:20:39.887459 11565 leader_election.cc:302] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 786f5f83aad243f5a9f6313c2c3681da, db14022bd7e64ec4873973f0166a1fc8; no voters: cf988718c8da472c975d2fa2b1e2a341
I20260810 06:20:39.887476 11252 leader_election.cc:302] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 786f5f83aad243f5a9f6313c2c3681da, db14022bd7e64ec4873973f0166a1fc8; no voters: cf988718c8da472c975d2fa2b1e2a341
I20260810 06:20:39.887637 12437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 06:20:39.887701 12437 raft_consensus.cc:482] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cf988718c8da472c975d2fa2b1e2a341)
I20260810 06:20:39.887740 12437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 1 FOLLOWER]: Advancing to term 2
I20260810 06:20:39.887817 12436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 06:20:39.887882 12436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 1 FOLLOWER]: Starting leader election (detected failure of leader cf988718c8da472c975d2fa2b1e2a341)
I20260810 06:20:39.887926 12436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 1 FOLLOWER]: Advancing to term 2
I20260810 06:20:39.889036 12436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } }
I20260810 06:20:39.889036 12437 raft_consensus.cc:504] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } }
I20260810 06:20:39.889380 12436 leader_election.cc:288] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [CANDIDATE]: Term 2 election: Requested vote from peers cf988718c8da472c975d2fa2b1e2a341 (127.31.204.254:34003), db14022bd7e64ec4873973f0166a1fc8 (127.31.204.252:40965)
I20260810 06:20:39.889392 12437 leader_election.cc:288] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [CANDIDATE]: Term 2 election: Requested vote from peers cf988718c8da472c975d2fa2b1e2a341 (127.31.204.254:34003), 786f5f83aad243f5a9f6313c2c3681da (127.31.204.253:36451)
I20260810 06:20:39.889787 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:39.889526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:51384)
0810 06:20:39.889647 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:39.889729 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 06:20:39.889889 11608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "786f5f83aad243f5a9f6313c2c3681da" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "db14022bd7e64ec4873973f0166a1fc8"
I20260810 06:20:39.889925 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:39.889589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:51398)
0810 06:20:39.889734 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:39.889787 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 06:20:39.890060 11608 raft_consensus.cc:2379] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 786f5f83aad243f5a9f6313c2c3681da in current term 2: Already voted for candidate db14022bd7e64ec4873973f0166a1fc8 in this term.
I20260810 06:20:39.889991 11289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db14022bd7e64ec4873973f0166a1fc8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "786f5f83aad243f5a9f6313c2c3681da"
I20260810 06:20:39.890192 11289 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate db14022bd7e64ec4873973f0166a1fc8 in current term 2: Already voted for candidate 786f5f83aad243f5a9f6313c2c3681da in this term.
I20260810 06:20:39.890635 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:39.890355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:51412)
0810 06:20:39.890529 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:39.890579 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260810 06:20:39.890717 11566 leader_election.cc:334] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cf988718c8da472c975d2fa2b1e2a341 (127.31.204.254:34003): Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
I20260810 06:20:39.890785 11566 leader_election.cc:302] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db14022bd7e64ec4873973f0166a1fc8; no voters: 786f5f83aad243f5a9f6313c2c3681da, cf988718c8da472c975d2fa2b1e2a341
I20260810 06:20:39.890908 12437 raft_consensus.cc:2726] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 06:20:39.890990 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:39.890775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:51424)
0810 06:20:39.890885 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:39.890936 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 06:20:39.891088 11251 leader_election.cc:334] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cf988718c8da472c975d2fa2b1e2a341 (127.31.204.254:34003): Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
I20260810 06:20:39.891183 11251 leader_election.cc:302] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 786f5f83aad243f5a9f6313c2c3681da; no voters: cf988718c8da472c975d2fa2b1e2a341, db14022bd7e64ec4873973f0166a1fc8
I20260810 06:20:39.891309 12436 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 06:20:40.369036 12452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:40.368618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.195:34417)
0810 06:20:40.368889 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:40.368947 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":92,"threads_started":1}
I20260810 06:20:40.371320 12455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:40.370865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.194:59147)
0810 06:20:40.371153 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:40.371228 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":79,"threads_started":1}
I20260810 06:20:40.371718 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:40.371173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.193:52301)
0810 06:20:40.371544 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:40.371607 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":77,"threads_started":1}
I20260810 06:20:41.369992 12457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.369610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.195:35451)
0810 06:20:41.369847 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:41.369904 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260810 06:20:41.372438 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.371931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.194:43301)
0810 06:20:41.372283 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:41.372344 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":67,"threads_started":1}
I20260810 06:20:41.372700 12462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.372300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.193:48543)
0810 06:20:41.372554 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:41.372614 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":80,"threads_started":1}
I20260810 06:20:41.898093 12465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:20:41.898238 12465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } }
I20260810 06:20:41.898690 12465 leader_election.cc:288] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cf988718c8da472c975d2fa2b1e2a341 (127.31.204.254:34003), db14022bd7e64ec4873973f0166a1fc8 (127.31.204.252:40965)
I20260810 06:20:41.899189 11608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "786f5f83aad243f5a9f6313c2c3681da" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "db14022bd7e64ec4873973f0166a1fc8" is_pre_election: true
I20260810 06:20:41.899299 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.898903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:51684)
0810 06:20:41.899117 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:41.899183 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":53,"threads_started":1}
I20260810 06:20:41.899353 11608 raft_consensus.cc:2454] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786f5f83aad243f5a9f6313c2c3681da in term 2.
I20260810 06:20:41.899657 11252 leader_election.cc:302] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786f5f83aad243f5a9f6313c2c3681da, db14022bd7e64ec4873973f0166a1fc8; no voters:
I20260810 06:20:41.899827 12465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 06:20:41.899896 12465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:20:41.899938 12465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 2 FOLLOWER]: Advancing to term 3
I20260810 06:20:41.900230 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.900035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:51698)
0810 06:20:41.900134 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:41.900180 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 06:20:41.900353 11251 leader_election.cc:334] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cf988718c8da472c975d2fa2b1e2a341 (127.31.204.254:34003): Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
I20260810 06:20:41.901063 12465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } }
I20260810 06:20:41.901357 12465 leader_election.cc:288] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [CANDIDATE]: Term 3 election: Requested vote from peers cf988718c8da472c975d2fa2b1e2a341 (127.31.204.254:34003), db14022bd7e64ec4873973f0166a1fc8 (127.31.204.252:40965)
I20260810 06:20:41.901671 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.901482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:51708)
0810 06:20:41.901578 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:41.901624 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 06:20:41.901684 11608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "786f5f83aad243f5a9f6313c2c3681da" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "db14022bd7e64ec4873973f0166a1fc8"
I20260810 06:20:41.901801 11608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 2 FOLLOWER]: Advancing to term 3
I20260810 06:20:41.902289 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.902100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:51716)
0810 06:20:41.902192 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:41.902237 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 06:20:41.902388 11251 leader_election.cc:334] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer cf988718c8da472c975d2fa2b1e2a341 (127.31.204.254:34003): Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
I20260810 06:20:41.902956 11608 raft_consensus.cc:2454] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786f5f83aad243f5a9f6313c2c3681da in term 3.
I20260810 06:20:41.903213 11252 leader_election.cc:302] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 786f5f83aad243f5a9f6313c2c3681da, db14022bd7e64ec4873973f0166a1fc8; no voters: cf988718c8da472c975d2fa2b1e2a341
I20260810 06:20:41.903369 12465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 3 FOLLOWER]: Leader election won for term 3
I20260810 06:20:41.903545 12465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [term 3 LEADER]: Becoming Leader. State: Replica: 786f5f83aad243f5a9f6313c2c3681da, State: Running, Role: LEADER
I20260810 06:20:41.903792 12465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } }
I20260810 06:20:41.904996 12473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 786f5f83aad243f5a9f6313c2c3681da. Latest consensus state: current_term: 3 leader_uuid: "786f5f83aad243f5a9f6313c2c3681da" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } } }
I20260810 06:20:41.905115 12473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:41.905411 12476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:41.907843 12476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:41.908223 12476 catalog_manager.cc:1259] Loaded cluster ID: a7dcab4da1e8485897cdfde29453d3a9
I20260810 06:20:41.908268 12476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:41.908810 12476 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:41.909170 12476 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: Loaded TSK: 0
I20260810 06:20:41.909868 12476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:20:41.926507 11272 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:20:41.929360 11272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:20:41.932091 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.931866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:51734)
0810 06:20:41.931982 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:41.932031 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 06:20:41.932137 11608 raft_consensus.cc:1261] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [term 3 FOLLOWER]: Refusing update from remote peer 786f5f83aad243f5a9f6313c2c3681da: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260810 06:20:41.932616 12465 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [LEADER]: Connected to new peer: Peer: permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 06:20:41.932744 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.932528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.0.0.1:51744)
0810 06:20:41.932632 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:41.932686 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 06:20:41.932844 11251 consensus_peers.cc:487] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da -> Peer cf988718c8da472c975d2fa2b1e2a341 (127.31.204.254:34003): Couldn't send request to peer cf988718c8da472c975d2fa2b1e2a341. Status: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 06:20:41.933961 12480 sys_catalog.cc:455] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 786f5f83aad243f5a9f6313c2c3681da. Latest consensus state: current_term: 3 leader_uuid: "786f5f83aad243f5a9f6313c2c3681da" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } } }
I20260810 06:20:41.934118 12480 sys_catalog.cc:458] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:20:41.935127 12465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "786f5f83aad243f5a9f6313c2c3681da" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } } }
I20260810 06:20:41.935240 12465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:41.935598 12480 sys_catalog.cc:455] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "786f5f83aad243f5a9f6313c2c3681da" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cf988718c8da472c975d2fa2b1e2a341" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34003 } } peers { permanent_uuid: "786f5f83aad243f5a9f6313c2c3681da" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 36451 } attrs { promote: false } } peers { permanent_uuid: "db14022bd7e64ec4873973f0166a1fc8" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 40965 } attrs { promote: false } } }
I20260810 06:20:41.935770 12480 sys_catalog.cc:458] T 00000000000000000000000000000000 P db14022bd7e64ec4873973f0166a1fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:20:41.944728 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.940413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:43569 (local address 127.0.0.1:42086)
0810 06:20:41.940900 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:41.940913 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:41.941008 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:41.941814 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:41.941817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:41.941836 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:41.941916 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:41.941922 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:41.942785 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:41.942788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:41.943533 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:41.943541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:41.944324 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:41.944327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:41.944420 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:41.944479 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 06:20:41.944544 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 06:20:41.944611 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":19}
I20260810 06:20:41.944747 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.940518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.31.204.195:43569)
0810 06:20:41.940746 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:41.940751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:41.941498 (+ 747us) server_negotiation.cc:407] Connection header received
0810 06:20:41.941562 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:41.941564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:41.941618 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:41.941698 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:41.942062 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:41.942671 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:41.943652 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:41.944177 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:41.944502 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:41.944550 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:41.944587 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 06:20:41.944647 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":66,"threads_started":1}
I20260810 06:20:41.945266 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.940657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:46103 (local address 127.0.0.1:49340)
0810 06:20:41.940918 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:41.940931 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:41.941119 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:41.941513 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:41.941515 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:41.941531 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:41.941639 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:41.941646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:41.942677 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:41.942680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:41.943926 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:41.943937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:41.944914 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:41.944918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:41.945006 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:41.945028 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:20:41.945087 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 06:20:41.945148 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":23,"thread_start_us":113,"threads_started":1}
I20260810 06:20:41.945230 12207 tablet_service.cc:1467] Processing CreateTablet for tablet a5b6329c9de0445ea30e746285f02fef (DEFAULT_TABLE table=test-table [id=0440daafeac748ca93fc39eabd370bba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:41.945542 12207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b6329c9de0445ea30e746285f02fef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:41.945655 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.940758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49340 (local address 127.31.204.194:46103)
0810 06:20:41.941169 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:41.941174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:41.941222 (+ 48us) server_negotiation.cc:407] Connection header received
0810 06:20:41.941272 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:41.941275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:41.941323 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:41.941395 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:41.941775 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:41.942555 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:41.944063 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:41.944678 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:41.944806 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:41.944856 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:41.945458 (+ 602us) server_negotiation.cc:299] Negotiation successful
0810 06:20:41.945532 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260810 06:20:41.946161 11992 tablet_service.cc:1467] Processing CreateTablet for tablet a5b6329c9de0445ea30e746285f02fef (DEFAULT_TABLE table=test-table [id=0440daafeac748ca93fc39eabd370bba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:41.946295 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.941204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:41705 (local address 127.0.0.1:55840)
0810 06:20:41.941802 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:41.941813 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:41.941896 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:41.942134 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:41.942136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:41.942154 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:41.942246 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:41.942251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:41.943157 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:41.943160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:41.944957 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:41.944964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:41.945896 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:41.945900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:41.945997 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:41.946048 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 06:20:41.946115 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 06:20:41.946177 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":22,"thread_start_us":100,"threads_started":1}
I20260810 06:20:41.946434 11992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b6329c9de0445ea30e746285f02fef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:41.946647 12494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.941360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55840 (local address 127.31.204.193:41705)
0810 06:20:41.941591 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:41.941595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:41.941893 (+ 298us) server_negotiation.cc:407] Connection header received
0810 06:20:41.941974 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:41.941977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:41.942045 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:41.942125 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:41.942347 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:41.943031 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:41.945069 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:41.945650 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:41.945771 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:41.945821 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:41.946443 (+ 622us) server_negotiation.cc:299] Negotiation successful
0810 06:20:41.946523 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260810 06:20:41.948071 12495 tablet_bootstrap.cc:492] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c: Bootstrap starting.
I20260810 06:20:41.948642 12498 tablet_bootstrap.cc:492] T a5b6329c9de0445ea30e746285f02fef P bc19037c918f422ba77c253d7c5eec17: Bootstrap starting.
I20260810 06:20:41.948973 12495 tablet_bootstrap.cc:654] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:41.949563 11776 tablet_service.cc:1467] Processing CreateTablet for tablet a5b6329c9de0445ea30e746285f02fef (DEFAULT_TABLE table=test-table [id=0440daafeac748ca93fc39eabd370bba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:41.949700 12498 tablet_bootstrap.cc:654] T a5b6329c9de0445ea30e746285f02fef P bc19037c918f422ba77c253d7c5eec17: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:41.949937 11776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b6329c9de0445ea30e746285f02fef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:41.950886 12498 tablet_bootstrap.cc:492] T a5b6329c9de0445ea30e746285f02fef P bc19037c918f422ba77c253d7c5eec17: No bootstrap required, opened a new log
I20260810 06:20:41.950975 12498 ts_tablet_manager.cc:1397] T a5b6329c9de0445ea30e746285f02fef P bc19037c918f422ba77c253d7c5eec17: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:41.951375 12498 raft_consensus.cc:348] T a5b6329c9de0445ea30e746285f02fef P bc19037c918f422ba77c253d7c5eec17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } } peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } }
I20260810 06:20:41.951520 12498 raft_consensus.cc:374] T a5b6329c9de0445ea30e746285f02fef P bc19037c918f422ba77c253d7c5eec17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:41.951571 12498 raft_consensus.cc:729] T a5b6329c9de0445ea30e746285f02fef P bc19037c918f422ba77c253d7c5eec17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc19037c918f422ba77c253d7c5eec17, State: Initialized, Role: FOLLOWER
I20260810 06:20:41.951740 12498 consensus_queue.cc:260] T a5b6329c9de0445ea30e746285f02fef P bc19037c918f422ba77c253d7c5eec17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } } peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } }
I20260810 06:20:41.952097 12498 ts_tablet_manager.cc:1428] T a5b6329c9de0445ea30e746285f02fef P bc19037c918f422ba77c253d7c5eec17: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:41.952714 12503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.952341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.194:44737)
0810 06:20:41.952562 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:41.952617 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":100,"threads_started":1}
I20260810 06:20:41.953030 12502 tablet_bootstrap.cc:492] T a5b6329c9de0445ea30e746285f02fef P e764aab3b1344241ac6bd8bc5639a237: Bootstrap starting.
I20260810 06:20:41.953810 12495 tablet_bootstrap.cc:492] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c: No bootstrap required, opened a new log
I20260810 06:20:41.953956 12495 ts_tablet_manager.cc:1397] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260810 06:20:41.954316 12502 tablet_bootstrap.cc:654] T a5b6329c9de0445ea30e746285f02fef P e764aab3b1344241ac6bd8bc5639a237: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:41.954463 12495 raft_consensus.cc:348] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } } peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } }
I20260810 06:20:41.954586 12102 heartbeater.cc:503] Master 127.31.204.253:36451 was elected leader, sending a full tablet report...
I20260810 06:20:41.954667 12495 raft_consensus.cc:374] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:41.954716 12495 raft_consensus.cc:729] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b59b0c145b014a6ea2444c33b301971c, State: Initialized, Role: FOLLOWER
I20260810 06:20:41.954849 12495 consensus_queue.cc:260] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } } peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } }
I20260810 06:20:41.955173 12495 ts_tablet_manager.cc:1428] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:41.956001 12503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.954622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.194:32895)
0810 06:20:41.955883 (+ 1261us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:41.955937 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1191}
I20260810 06:20:41.956177 12502 tablet_bootstrap.cc:492] T a5b6329c9de0445ea30e746285f02fef P e764aab3b1344241ac6bd8bc5639a237: No bootstrap required, opened a new log
I20260810 06:20:41.956280 12508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.955583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.195:42959)
0810 06:20:41.956057 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:41.956153 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":88,"threads_started":1}
I20260810 06:20:41.956315 12502 ts_tablet_manager.cc:1397] T a5b6329c9de0445ea30e746285f02fef P e764aab3b1344241ac6bd8bc5639a237: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 06:20:41.956707 12502 raft_consensus.cc:348] T a5b6329c9de0445ea30e746285f02fef P e764aab3b1344241ac6bd8bc5639a237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } } peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } }
I20260810 06:20:41.956866 12502 raft_consensus.cc:374] T a5b6329c9de0445ea30e746285f02fef P e764aab3b1344241ac6bd8bc5639a237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:41.956945 12508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.956707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.195:58997)
0810 06:20:41.956842 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:41.956891 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 06:20:41.956935 12502 raft_consensus.cc:729] T a5b6329c9de0445ea30e746285f02fef P e764aab3b1344241ac6bd8bc5639a237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e764aab3b1344241ac6bd8bc5639a237, State: Initialized, Role: FOLLOWER
I20260810 06:20:41.957064 12502 consensus_queue.cc:260] T a5b6329c9de0445ea30e746285f02fef P e764aab3b1344241ac6bd8bc5639a237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } } peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } }
W20260810 06:20:41.957134 12319 tablet.cc:2369] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:20:41.957255 12318 heartbeater.cc:503] Master 127.31.204.253:36451 was elected leader, sending a full tablet report...
I20260810 06:20:41.957399 12502 ts_tablet_manager.cc:1428] T a5b6329c9de0445ea30e746285f02fef P e764aab3b1344241ac6bd8bc5639a237: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:41.958214 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.957738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.193:42527)
0810 06:20:41.958073 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:41.958127 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":88,"threads_started":1}
I20260810 06:20:41.959043 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:41.958650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.193:56391)
0810 06:20:41.958929 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:41.958976 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":106,"threads_started":1}
I20260810 06:20:41.959142 11877 heartbeater.cc:503] Master 127.31.204.253:36451 was elected leader, sending a full tablet report...
W20260810 06:20:41.978461 12103 tablet.cc:2369] T a5b6329c9de0445ea30e746285f02fef P bc19037c918f422ba77c253d7c5eec17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:20:42.011463 12506 raft_consensus.cc:482] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:20:42.011601 12506 raft_consensus.cc:504] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } } peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } }
I20260810 06:20:42.012627 12506 leader_election.cc:288] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc19037c918f422ba77c253d7c5eec17 (127.31.204.194:46103), e764aab3b1344241ac6bd8bc5639a237 (127.31.204.193:41705)
I20260810 06:20:42.017311 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:42.012958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:45039 (local address 127.31.204.194:46103)
0810 06:20:42.013083 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:42.013088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:42.013453 (+ 365us) server_negotiation.cc:407] Connection header received
0810 06:20:42.013555 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:42.013557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:42.013604 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:42.013681 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:42.013886 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:42.014612 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:42.016115 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:42.016702 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:42.016799 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:42.016842 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:42.017115 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 06:20:42.017186 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 06:20:42.017313 12494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:42.012954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:46029 (local address 127.31.204.193:41705)
0810 06:20:42.013085 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:42.013090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:42.013107 (+ 17us) server_negotiation.cc:407] Connection header received
0810 06:20:42.013192 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:42.013195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:42.013261 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:42.013343 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:42.013636 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:42.014440 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:42.016049 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:42.016655 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:42.016791 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:42.016842 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:42.017115 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 06:20:42.017186 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 06:20:42.017338 12508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:42.012838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:41705 (local address 127.31.204.195:46029)
0810 06:20:42.012981 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:42.012997 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:42.013099 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:42.013370 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:42.013373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:42.013395 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:42.013520 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:42.013527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:42.014558 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:42.014561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:42.015913 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:42.015924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:42.016796 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:42.016801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:42.016960 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:42.017012 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 06:20:42.017090 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 06:20:42.017186 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":70}
I20260810 06:20:42.017349 12516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:42.012838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:46103 (local address 127.31.204.195:45039)
0810 06:20:42.013372 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:42.013384 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:42.013468 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:42.013691 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:42.013693 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:42.013709 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:42.013786 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:42.013790 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:42.014702 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:42.014705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:42.015949 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:42.015965 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:42.016815 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:42.016819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:42.016966 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:42.017013 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 06:20:42.017090 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 06:20:42.017186 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":335,"thread_start_us":145,"threads_started":1}
I20260810 06:20:42.017844 11805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b6329c9de0445ea30e746285f02fef" candidate_uuid: "b59b0c145b014a6ea2444c33b301971c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e764aab3b1344241ac6bd8bc5639a237" is_pre_election: true
I20260810 06:20:42.017887 12020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b6329c9de0445ea30e746285f02fef" candidate_uuid: "b59b0c145b014a6ea2444c33b301971c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc19037c918f422ba77c253d7c5eec17" is_pre_election: true
I20260810 06:20:42.018003 12020 raft_consensus.cc:2454] T a5b6329c9de0445ea30e746285f02fef P bc19037c918f422ba77c253d7c5eec17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b59b0c145b014a6ea2444c33b301971c in term 0.
I20260810 06:20:42.018083 11805 raft_consensus.cc:2454] T a5b6329c9de0445ea30e746285f02fef P e764aab3b1344241ac6bd8bc5639a237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b59b0c145b014a6ea2444c33b301971c in term 0.
I20260810 06:20:42.018420 12140 leader_election.cc:302] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b59b0c145b014a6ea2444c33b301971c, bc19037c918f422ba77c253d7c5eec17, e764aab3b1344241ac6bd8bc5639a237; no voters:
I20260810 06:20:42.018692 12506 raft_consensus.cc:2781] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:20:42.018747 12506 raft_consensus.cc:482] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:20:42.018783 12506 raft_consensus.cc:3037] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:42.019769 12506 raft_consensus.cc:504] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } } peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } }
I20260810 06:20:42.020080 12506 leader_election.cc:288] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [CANDIDATE]: Term 1 election: Requested vote from peers bc19037c918f422ba77c253d7c5eec17 (127.31.204.194:46103), e764aab3b1344241ac6bd8bc5639a237 (127.31.204.193:41705)
I20260810 06:20:42.020378 11805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b6329c9de0445ea30e746285f02fef" candidate_uuid: "b59b0c145b014a6ea2444c33b301971c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e764aab3b1344241ac6bd8bc5639a237"
I20260810 06:20:42.020382 12020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b6329c9de0445ea30e746285f02fef" candidate_uuid: "b59b0c145b014a6ea2444c33b301971c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc19037c918f422ba77c253d7c5eec17"
I20260810 06:20:42.020473 11805 raft_consensus.cc:3037] T a5b6329c9de0445ea30e746285f02fef P e764aab3b1344241ac6bd8bc5639a237 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:42.020483 12020 raft_consensus.cc:3037] T a5b6329c9de0445ea30e746285f02fef P bc19037c918f422ba77c253d7c5eec17 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:42.021363 12020 raft_consensus.cc:2454] T a5b6329c9de0445ea30e746285f02fef P bc19037c918f422ba77c253d7c5eec17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b59b0c145b014a6ea2444c33b301971c in term 1.
I20260810 06:20:42.021368 11805 raft_consensus.cc:2454] T a5b6329c9de0445ea30e746285f02fef P e764aab3b1344241ac6bd8bc5639a237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b59b0c145b014a6ea2444c33b301971c in term 1.
I20260810 06:20:42.021611 12140 leader_election.cc:302] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b59b0c145b014a6ea2444c33b301971c, bc19037c918f422ba77c253d7c5eec17; no voters:
I20260810 06:20:42.021770 12506 raft_consensus.cc:2781] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:42.021984 12506 raft_consensus.cc:686] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [term 1 LEADER]: Becoming Leader. State: Replica: b59b0c145b014a6ea2444c33b301971c, State: Running, Role: LEADER
I20260810 06:20:42.022280 12506 consensus_queue.cc:237] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } } peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } }
I20260810 06:20:42.022742 12516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:42.022495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.195:41149)
0810 06:20:42.022606 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:42.022672 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260810 06:20:42.024446 11272 catalog_manager.cc:5495] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c reported cstate change: term changed from 0 to 1, leader changed from <none> to b59b0c145b014a6ea2444c33b301971c (127.31.204.195). New cstate: current_term: 1 leader_uuid: "b59b0c145b014a6ea2444c33b301971c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b59b0c145b014a6ea2444c33b301971c" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 } health_report { overall_health: UNKNOWN } } }
W20260810 06:20:42.031141 11881 tablet.cc:2369] T a5b6329c9de0445ea30e746285f02fef P e764aab3b1344241ac6bd8bc5639a237: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:20:42.041910 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:42.038591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.31.204.195:43569)
0810 06:20:42.038709 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:42.038713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:42.038725 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:20:42.038866 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:42.038868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:42.038918 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:42.039001 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:42.039236 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:42.039760 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:42.040585 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:42.040810 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:42.040899 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:42.041129 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:42.041550 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:42.041648 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:42.041740 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 06:20:42.041794 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:20:42.044504 11805 raft_consensus.cc:1261] T a5b6329c9de0445ea30e746285f02fef P e764aab3b1344241ac6bd8bc5639a237 [term 1 FOLLOWER]: Refusing update from remote peer b59b0c145b014a6ea2444c33b301971c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:42.044988 12020 raft_consensus.cc:1261] T a5b6329c9de0445ea30e746285f02fef P bc19037c918f422ba77c253d7c5eec17 [term 1 FOLLOWER]: Refusing update from remote peer b59b0c145b014a6ea2444c33b301971c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:42.045081 12517 consensus_queue.cc:1035] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e764aab3b1344241ac6bd8bc5639a237" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:42.045400 12506 consensus_queue.cc:1035] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc19037c918f422ba77c253d7c5eec17" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:42.047492 12514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:42.047254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.193:47777)
0810 06:20:42.047350 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:42.047407 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 06:20:42.047781 12503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:42.047494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.194:50547)
0810 06:20:42.047642 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:42.047687 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 06:20:42.049886 12522 mvcc.cc:205] Tried to move back new op lower bound from 7316860281009238016 to 7316860280926023680. Current Snapshot: MvccSnapshot[applied={T|T < 7316860281009238016}]
I20260810 06:20:42.050046 12516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:42.049756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.195:53373)
0810 06:20:42.049902 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:42.049955 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 06:20:42.050581 12514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:42.050345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.193:32935)
0810 06:20:42.050466 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:42.050520 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 06:20:42.053586 12516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:42.053397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.195:56291)
0810 06:20:42.053480 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:42.053528 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260810 06:20:42.054898 12521 mvcc.cc:205] Tried to move back new op lower bound from 7316860281009238016 to 7316860280926023680. Current Snapshot: MvccSnapshot[applied={T|T < 7316860281009238016 or (T in {7316860281009238016})}]
I20260810 06:20:42.055258 12503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:42.054586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34003 (local address 127.31.204.194:51711)
0810 06:20:42.055138 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:42.055192 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483}
I20260810 06:20:42.059051 11272 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:42.059183 11272 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:42.062501 11272 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 786f5f83aad243f5a9f6313c2c3681da: Sending DeleteTablet for 3 replicas of tablet a5b6329c9de0445ea30e746285f02fef
I20260810 06:20:42.063241 12207 tablet_service.cc:1514] Processing DeleteTablet for tablet a5b6329c9de0445ea30e746285f02fef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:42 UTC) from {username='oryx', principal='oryx/127.31.204.253@KRBTEST.COM'} at 127.0.0.1:42086
I20260810 06:20:42.063271 11776 tablet_service.cc:1514] Processing DeleteTablet for tablet a5b6329c9de0445ea30e746285f02fef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:42 UTC) from {username='oryx', principal='oryx/127.31.204.253@KRBTEST.COM'} at 127.0.0.1:55840
I20260810 06:20:42.063706 11992 tablet_service.cc:1514] Processing DeleteTablet for tablet a5b6329c9de0445ea30e746285f02fef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:42 UTC) from {username='oryx', principal='oryx/127.31.204.253@KRBTEST.COM'} at 127.0.0.1:49340
I20260810 06:20:42.064217 12534 tablet_replica.cc:330] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c: stopping tablet replica
I20260810 06:20:42.064364 12534 raft_consensus.cc:2229] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:20:42.064584 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 11522
I20260810 06:20:42.067274 12534 raft_consensus.cc:2258] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:42.072512 12534 ts_tablet_manager.cc:1905] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:42.075230 12536 tablet_replica.cc:330] T a5b6329c9de0445ea30e746285f02fef P bc19037c918f422ba77c253d7c5eec17: stopping tablet replica
I20260810 06:20:42.075374 12536 raft_consensus.cc:2229] T a5b6329c9de0445ea30e746285f02fef P bc19037c918f422ba77c253d7c5eec17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:42.075501 12536 raft_consensus.cc:2258] T a5b6329c9de0445ea30e746285f02fef P bc19037c918f422ba77c253d7c5eec17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:42.076187 12534 ts_tablet_manager.cc:1918] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:20:42.076269 12534 log.cc:1196] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/a5b6329c9de0445ea30e746285f02fef
I20260810 06:20:42.076555 12534 ts_tablet_manager.cc:1939] T a5b6329c9de0445ea30e746285f02fef P b59b0c145b014a6ea2444c33b301971c: Deleting consensus metadata
I20260810 06:20:42.077315 11249 catalog_manager.cc:4841] TS b59b0c145b014a6ea2444c33b301971c (127.31.204.195:43569): tablet a5b6329c9de0445ea30e746285f02fef (table test-table [id=0440daafeac748ca93fc39eabd370bba]) successfully deleted
I20260810 06:20:42.084115 12536 ts_tablet_manager.cc:1905] T a5b6329c9de0445ea30e746285f02fef P bc19037c918f422ba77c253d7c5eec17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 06:20:42.085655 12492 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:42.085345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:41705 (local address 127.0.0.1:55846)
0810 06:20:42.085482 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:42.085546 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:41705: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 06:20:42.085788 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 11888
W20260810 06:20:42.088719 11249 catalog_manager.cc:4584] TS e764aab3b1344241ac6bd8bc5639a237 (127.31.204.193:41705): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a5b6329c9de0445ea30e746285f02fef: Network error: Client connection negotiation failed: client connection to 127.31.204.193:41705: connect: Connection reset by peer (error 104)
I20260810 06:20:42.097373 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 12110
I20260810 06:20:42.097654 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:42.097404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:46103 (local address 127.0.0.1:49356)
0810 06:20:42.097528 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:42.097577 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.194:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 06:20:42.107759 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 11031
I20260810 06:20:42.119894 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 11229
2026-08-10T06:20:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7324 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:20:42 dist-test-slave-q9h9 krb5kdc[12541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:20:42 dist-test-slave-q9h9 krb5kdc[12541](info): set up 2 sockets
Aug 10 06:20:42 dist-test-slave-q9h9 krb5kdc[12541](info): commencing operation
krb5kdc: starting...
W20260810 06:20:43.304306 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:20:43 dist-test-slave-q9h9 krb5kdc[12541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:20:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:20:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.254.keytab.
Entry for principal oryx/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.254.keytab.
I20260810 06:20:43.414463 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:43157
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.254.keytab
--principal=oryx/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:43157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:43.519125 12567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:43.519358 12567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:43.519411 12567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:43.523229 12567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:43.523274 12567 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 06:20:43.523296 12567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:43.523319 12567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:43.523340 12567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:43.523368 12567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:20:43.528296 12567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:43157
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.254.keytab
--principal=oryx/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:43157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:43.529150 12567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:43.530069 12567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:43.537671 12583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:43.537644 12580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:43.537647 12578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:43.537752 12567 server_base.cc:1034] running on GCE node
I20260810 06:20:43.538440 12567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:43.539424 12567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:43.540571 12567 hybrid_clock.cc:648] HybridClock initialized: now 1786342843540555 us; error 34 us; skew 500 ppm
Aug 10 06:20:43 dist-test-slave-q9h9 krb5kdc[12541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342843, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:43.543244 12567 init.cc:377] Logged in from keytab as oryx/127.31.204.254@KRBTEST.COM (short username oryx)
I20260810 06:20:43.544032 12567 webserver.cc:460] Webserver started at http://127.31.204.254:38023/ using document root <none> and password file <none>
I20260810 06:20:43.544516 12567 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:43.544580 12567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:43.544756 12567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:43.546648 12567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "0e75c0fa81ae438f8516c8511f91aa9c"
format_stamp: "Formatted at 2026-08-10 06:20:43 on dist-test-slave-q9h9"
server_key: "5cd2e5e0baeef530077bb96818f9b6a2"
server_key_iv: "dd6611e34841e4363fadb0b842ef65cc"
server_key_version: "encryptionkey@0"
I20260810 06:20:43.547148 12567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "0e75c0fa81ae438f8516c8511f91aa9c"
format_stamp: "Formatted at 2026-08-10 06:20:43 on dist-test-slave-q9h9"
server_key: "5cd2e5e0baeef530077bb96818f9b6a2"
server_key_iv: "dd6611e34841e4363fadb0b842ef65cc"
server_key_version: "encryptionkey@0"
I20260810 06:20:43.550450 12567 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:20:43.552587 12592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:43.553689 12567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:43.553798 12567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "0e75c0fa81ae438f8516c8511f91aa9c"
format_stamp: "Formatted at 2026-08-10 06:20:43 on dist-test-slave-q9h9"
server_key: "5cd2e5e0baeef530077bb96818f9b6a2"
server_key_iv: "dd6611e34841e4363fadb0b842ef65cc"
server_key_version: "encryptionkey@0"
I20260810 06:20:43.553884 12567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:43.567251 12567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:43.569973 12567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:43.570173 12567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:43.579983 12567 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:43157
I20260810 06:20:43.580008 12683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:43157 every 8 connection(s)
I20260810 06:20:43.580842 12567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:20:43.583427 12687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:43.588619 12687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c: Bootstrap starting.
I20260810 06:20:43.589432 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 12567
I20260810 06:20:43.589555 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:20:43.589823 32563 external_mini_cluster.cc:1428] Setting key 76f8cfca90c4df1a2d51934232d39c88
I20260810 06:20:43.591157 12687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:43.591857 12687 log.cc:824] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:43.593889 12687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c: No bootstrap required, opened a new log
I20260810 06:20:43.597967 12687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e75c0fa81ae438f8516c8511f91aa9c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43157 } }
I20260810 06:20:43.598223 12687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:43.598289 12687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e75c0fa81ae438f8516c8511f91aa9c, State: Initialized, Role: FOLLOWER
Aug 10 06:20:43 dist-test-slave-q9h9 krb5kdc[12541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM
I20260810 06:20:43.598693 12687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e75c0fa81ae438f8516c8511f91aa9c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43157 } }
I20260810 06:20:43.598824 12687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:20:43.598889 12687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:20:43.598981 12687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:43.599900 12687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e75c0fa81ae438f8516c8511f91aa9c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43157 } }
I20260810 06:20:43.600206 12687 leader_election.cc:302] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e75c0fa81ae438f8516c8511f91aa9c; no voters:
I20260810 06:20:43.600476 12687 leader_election.cc:288] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:20:43.600586 12698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:43.600800 12698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c [term 1 LEADER]: Becoming Leader. State: Replica: 0e75c0fa81ae438f8516c8511f91aa9c, State: Running, Role: LEADER
I20260810 06:20:43.601161 12698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e75c0fa81ae438f8516c8511f91aa9c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43157 } }
I20260810 06:20:43.601665 12687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:43.602850 12700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e75c0fa81ae438f8516c8511f91aa9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e75c0fa81ae438f8516c8511f91aa9c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43157 } } }
I20260810 06:20:43.602919 12701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e75c0fa81ae438f8516c8511f91aa9c. Latest consensus state: current_term: 1 leader_uuid: "0e75c0fa81ae438f8516c8511f91aa9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e75c0fa81ae438f8516c8511f91aa9c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43157 } } }
I20260810 06:20:43.603058 12700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:43.603065 12701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:43.603474 12713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:43.605185 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:43.590845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44214 (local address 127.31.204.254:43157)
0810 06:20:43.591247 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:43.591265 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:43.591307 (+ 42us) server_negotiation.cc:407] Connection header received
0810 06:20:43.591920 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:43.591936 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:43.592312 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:43.592650 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:43.593488 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:43.594492 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:43.595677 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:43.596108 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:43.596223 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:43.598964 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:43.598971 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:43.598986 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:43.599016 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:43.600661 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:43.601148 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:43.601154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:43.601159 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:43.601298 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:43.601588 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:43.601591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:43.601593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:43.602133 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:43.602312 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:43.603932 (+ 1620us) server_negotiation.cc:299] Negotiation successful
0810 06:20:43.604222 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":109,"threads_started":1}
I20260810 06:20:43.606704 12713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:43.612516 12713 catalog_manager.cc:1347] Generated new cluster ID: 870ce8e8d53c488abba4fbfc2b286b1b
I20260810 06:20:43.612582 12713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:43.623687 12713 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:20:43.624524 12713 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:43.633868 12713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c: Generated new TSK 0
I20260810 06:20:43.634553 12713 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.193.keytab.
Entry for principal oryx/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.193.keytab.
I20260810 06:20:43.662842 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:43157
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.193.keytab
--principal=oryx/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:43.770887 12736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:43.771121 12736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:43.771168 12736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:43.775141 12736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:43.775197 12736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:43.775239 12736 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 06:20:43.775290 12736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:20:43.780463 12736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.193.keytab
--principal=oryx/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.204.254:43157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:43.781427 12736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:43.782464 12736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:43.791445 12742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:43.791489 12741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:43.791467 12736 server_base.cc:1034] running on GCE node
W20260810 06:20:43.791652 12744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:43.792014 12736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:43.792616 12736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:43.793763 12736 hybrid_clock.cc:648] HybridClock initialized: now 1786342843793747 us; error 41 us; skew 500 ppm
Aug 10 06:20:43 dist-test-slave-q9h9 krb5kdc[12541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342843, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:43.796334 12736 init.cc:377] Logged in from keytab as oryx/127.31.204.193@KRBTEST.COM (short username oryx)
I20260810 06:20:43.797128 12736 webserver.cc:460] Webserver started at http://127.31.204.193:39353/ using document root <none> and password file <none>
I20260810 06:20:43.797637 12736 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:43.797698 12736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:43.797859 12736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:43.799875 12736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "da22030ce30d4474b5a0043fccea9bf5"
format_stamp: "Formatted at 2026-08-10 06:20:43 on dist-test-slave-q9h9"
server_key: "e2648c6e5ea2d051b58cfc3f8968ee2d"
server_key_iv: "07cc81ecdf013d5d3bd771a0d96e206e"
server_key_version: "encryptionkey@0"
I20260810 06:20:43.800365 12736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "da22030ce30d4474b5a0043fccea9bf5"
format_stamp: "Formatted at 2026-08-10 06:20:43 on dist-test-slave-q9h9"
server_key: "e2648c6e5ea2d051b58cfc3f8968ee2d"
server_key_iv: "07cc81ecdf013d5d3bd771a0d96e206e"
server_key_version: "encryptionkey@0"
I20260810 06:20:43.803997 12736 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:20:43.806144 12753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:43.807238 12736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:43.807328 12736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "da22030ce30d4474b5a0043fccea9bf5"
format_stamp: "Formatted at 2026-08-10 06:20:43 on dist-test-slave-q9h9"
server_key: "e2648c6e5ea2d051b58cfc3f8968ee2d"
server_key_iv: "07cc81ecdf013d5d3bd771a0d96e206e"
server_key_version: "encryptionkey@0"
I20260810 06:20:43.807404 12736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:43.816998 12736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:43.819972 12736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:43.820109 12736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:43.821372 12736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:43.821415 12736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:43.821451 12736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:43.821465 12736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 06:20:43 dist-test-slave-q9h9 krb5kdc[12541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342843, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM
I20260810 06:20:43.831555 12736 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:42859
I20260810 06:20:43.831640 12949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:42859 every 8 connection(s)
I20260810 06:20:43.832542 12736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:20:43.833797 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:43.822140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:60453 (local address 127.31.204.254:43157)
0810 06:20:43.822402 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:43.822407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:43.823185 (+ 778us) server_negotiation.cc:407] Connection header received
0810 06:20:43.824158 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:43.824161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:43.824215 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:43.824257 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:43.825477 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:43.826177 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:43.827257 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:43.827495 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:43.827583 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:43.830375 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:43.830379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:43.830382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:43.830409 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:43.831753 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:43.832322 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:43.832325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:43.832326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:43.832401 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:43.832751 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:43.832754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:43.832755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:43.832960 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:43.833054 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:43.833572 (+ 518us) server_negotiation.cc:299] Negotiation successful
0810 06:20:43.833675 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260810 06:20:43.834519 12773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:43.822569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:43157 (local address 127.31.204.193:60453)
0810 06:20:43.823016 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:43.823062 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:43.823930 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:43.824562 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:43.824568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:43.824855 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:43.825332 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:43.825347 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:43.826311 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:43.826315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:43.827139 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:43.827149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:43.827650 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:43.827655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:43.827845 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:43.828402 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:43.828421 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:43.830199 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:43.831888 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:43.831893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:43.831905 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:43.832217 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:43.832488 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:43.832491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:43.832493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:43.832660 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:43.833105 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:43.833111 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:43.833371 (+ 260us) client_negotiation.cc:769] Sending connection context
0810 06:20:43.833517 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 06:20:43.833740 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":114,"threads_started":1}
I20260810 06:20:43.835678 12951 heartbeater.cc:347] Connected to a master server at 127.31.204.254:43157
I20260810 06:20:43.835815 12951 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:43.836328 12951 heartbeater.cc:511] Master 127.31.204.254:43157 requested a full tablet report, sending...
I20260810 06:20:43.837764 12617 ts_manager.cc:194] Registered new tserver with Master: da22030ce30d4474b5a0043fccea9bf5 (127.31.204.193:42859)
I20260810 06:20:43.838621 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 12736
I20260810 06:20:43.838747 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:20:43.839087 32563 external_mini_cluster.cc:1428] Setting key c84ea6447488fa7b9fa6d615a342c407
I20260810 06:20:43.839195 12617 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:60453
I20260810 06:20:43.844758 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:43.837802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:37793 (local address 127.31.204.254:43157)
0810 06:20:43.837905 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:43.837908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:43.838131 (+ 223us) server_negotiation.cc:407] Connection header received
0810 06:20:43.838255 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:43.838257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:43.838293 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:43.838351 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:43.838926 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:43.839572 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:43.840614 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:43.840870 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:43.840933 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:43.842077 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:43.842081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:43.842083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:43.842108 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:43.843385 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:43.843815 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:43.843818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:43.843819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:43.843880 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:43.844116 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:43.844118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:43.844119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:43.844264 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:43.844334 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:43.844575 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 06:20:43.844656 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 06:20:43.845067 12958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:43.837678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:43157 (local address 127.31.204.193:37793)
0810 06:20:43.838065 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:43.838080 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:43.838168 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:43.838385 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:43.838388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:43.838589 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:43.838752 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:43.838761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:43.839690 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:43.839693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:43.840490 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:43.840502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:43.841067 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:43.841070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:43.841149 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:43.841583 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:43.841598 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:43.841925 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:43.843495 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:43.843498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:43.843500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:43.843733 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:43.843953 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:43.843956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:43.843957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:43.844043 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:43.844349 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:43.844352 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:43.844440 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 06:20:43.844545 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 06:20:43.844670 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for oryx/127.31.204.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.204.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.194.keytab.
Entry for principal oryx/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.194.keytab.
I20260810 06:20:43.866429 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:43157
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.194.keytab
--principal=oryx/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:43.972644 12963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:43.972946 12963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:43.973021 12963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:43.977216 12963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:43.977269 12963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:43.977305 12963 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 06:20:43.977342 12963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:20:43.982372 12963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.194.keytab
--principal=oryx/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.204.254:43157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:43.983337 12963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:43.984298 12963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:43.993469 12972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:43.993533 12963 server_base.cc:1034] running on GCE node
W20260810 06:20:43.993490 12977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:43.993482 12975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:43.994167 12963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:43.994753 12963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:43.995924 12963 hybrid_clock.cc:648] HybridClock initialized: now 1786342843995909 us; error 27 us; skew 500 ppm
Aug 10 06:20:43 dist-test-slave-q9h9 krb5kdc[12541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342843, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:43.998533 12963 init.cc:377] Logged in from keytab as oryx/127.31.204.194@KRBTEST.COM (short username oryx)
I20260810 06:20:43.999307 12963 webserver.cc:460] Webserver started at http://127.31.204.194:33039/ using document root <none> and password file <none>
I20260810 06:20:43.999809 12963 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:43.999872 12963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:44.000053 12963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:44.002067 12963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "593684227d124bdda7254e54ed161e13"
format_stamp: "Formatted at 2026-08-10 06:20:43 on dist-test-slave-q9h9"
server_key: "eb1d9e8a2b8ac4fc89b96ae690309584"
server_key_iv: "0cf6cbc76f1d372b27681dd8222553c9"
server_key_version: "encryptionkey@0"
I20260810 06:20:44.002627 12963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "593684227d124bdda7254e54ed161e13"
format_stamp: "Formatted at 2026-08-10 06:20:43 on dist-test-slave-q9h9"
server_key: "eb1d9e8a2b8ac4fc89b96ae690309584"
server_key_iv: "0cf6cbc76f1d372b27681dd8222553c9"
server_key_version: "encryptionkey@0"
I20260810 06:20:44.006170 12963 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:20:44.008327 12987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:44.009407 12963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:44.009519 12963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "593684227d124bdda7254e54ed161e13"
format_stamp: "Formatted at 2026-08-10 06:20:43 on dist-test-slave-q9h9"
server_key: "eb1d9e8a2b8ac4fc89b96ae690309584"
server_key_iv: "0cf6cbc76f1d372b27681dd8222553c9"
server_key_version: "encryptionkey@0"
I20260810 06:20:44.009604 12963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:44.038151 12963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:44.041110 12963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:44.041285 12963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:44.042742 12963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:44.042793 12963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:44.042846 12963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:44.042871 12963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 06:20:44 dist-test-slave-q9h9 krb5kdc[12541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342843, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM
I20260810 06:20:44.053095 12963 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:46381
I20260810 06:20:44.053166 13170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:46381 every 8 connection(s)
I20260810 06:20:44.054230 12963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:20:44.056586 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.043575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:38489 (local address 127.31.204.254:43157)
0810 06:20:44.043710 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:44.043715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:44.044694 (+ 979us) server_negotiation.cc:407] Connection header received
0810 06:20:44.045760 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:44.045763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:44.045822 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:44.045902 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:44.047254 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.048042 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.049256 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.049570 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.049675 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.052831 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:44.052835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:44.052837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:44.052867 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:44.054407 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:44.054990 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:44.054993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:44.054995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:44.055093 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:44.055517 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:44.055521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:44.055523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:44.055731 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:44.055811 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:44.056364 (+ 553us) server_negotiation.cc:299] Negotiation successful
0810 06:20:44.056468 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 06:20:44.057540 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 12963
I20260810 06:20:44.057569 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.044040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:43157 (local address 127.31.204.194:38489)
0810 06:20:44.044528 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:44.044575 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:44.045539 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:44.046232 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:44.046241 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:44.046589 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:44.047095 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.047109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.048207 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.048211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.049127 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.049137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.049839 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.049842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.049974 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.051047 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:44.051066 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:44.052650 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:44.054519 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:44.054525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:44.054539 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:44.054894 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:44.055217 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:44.055220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:44.055223 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:44.055406 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:44.055833 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:44.055839 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:44.056124 (+ 285us) client_negotiation.cc:769] Sending connection context
0810 06:20:44.056329 (+ 205us) client_negotiation.cc:240] Negotiation successful
0810 06:20:44.056632 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":134,"threads_started":1}
I20260810 06:20:44.057672 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:20:44.058061 32563 external_mini_cluster.cc:1428] Setting key c137b4a001a0eed6a39340ccba1abfae
I20260810 06:20:44.059192 13171 heartbeater.cc:347] Connected to a master server at 127.31.204.254:43157
I20260810 06:20:44.059340 13171 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:44.059813 13171 heartbeater.cc:511] Master 127.31.204.254:43157 requested a full tablet report, sending...
I20260810 06:20:44.060748 12615 ts_manager.cc:194] Registered new tserver with Master: 593684227d124bdda7254e54ed161e13 (127.31.204.194:46381)
I20260810 06:20:44.061419 12615 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:38489
WARNING: no policy specified for oryx/127.31.204.195@KRBTEST.COM; defaulting to no policy
I20260810 06:20:44.069592 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.061367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:52331 (local address 127.31.204.254:43157)
0810 06:20:44.061476 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:44.061480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:44.061732 (+ 252us) server_negotiation.cc:407] Connection header received
0810 06:20:44.061892 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:44.061895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:44.061939 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:44.061985 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:44.062617 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.063322 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.064323 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.064655 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.064756 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.065998 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:44.066002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:44.066005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:44.066058 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:44.067670 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:44.068143 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:44.068146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:44.068148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:44.068230 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:44.068560 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:44.068563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:44.068565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:44.068835 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:44.068909 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:44.069329 (+ 420us) server_negotiation.cc:299] Negotiation successful
0810 06:20:44.069439 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 06:20:44.070120 13185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.061356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:43157 (local address 127.31.204.194:52331)
0810 06:20:44.061658 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:44.061670 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:44.061794 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:44.062138 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:44.062141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:44.062361 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:44.062487 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.062494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.063443 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.063446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.064197 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.064209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.064789 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.064793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.064910 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.065467 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:44.065483 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:44.065865 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:44.067794 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:44.067797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:44.067799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:44.068032 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:44.068355 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:44.068358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:44.068360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:44.068455 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:44.068986 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:44.068989 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:44.069079 (+ 90us) client_negotiation.cc:769] Sending connection context
0810 06:20:44.069736 (+ 657us) client_negotiation.cc:240] Negotiation successful
0810 06:20:44.069915 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.204.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.195.keytab.
Entry for principal oryx/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.195.keytab.
I20260810 06:20:44.086539 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:43157
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.195.keytab
--principal=oryx/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:44.207964 13189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:44.208182 13189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:44.208222 13189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:44.212134 13189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:44.212186 13189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:44.212237 13189 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 06:20:44.212281 13189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:20:44.217208 13189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.195.keytab
--principal=oryx/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.204.254:43157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:44.218112 13189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:44.219029 13189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:44.228147 13198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:44.228154 13194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:44.228238 13189 server_base.cc:1034] running on GCE node
W20260810 06:20:44.228154 13195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:44.228735 13189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:44.229297 13189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:44.230446 13189 hybrid_clock.cc:648] HybridClock initialized: now 1786342844230431 us; error 32 us; skew 500 ppm
Aug 10 06:20:44 dist-test-slave-q9h9 krb5kdc[12541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342844, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:44.232939 13189 init.cc:377] Logged in from keytab as oryx/127.31.204.195@KRBTEST.COM (short username oryx)
I20260810 06:20:44.233744 13189 webserver.cc:460] Webserver started at http://127.31.204.195:45371/ using document root <none> and password file <none>
I20260810 06:20:44.234284 13189 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:44.234345 13189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:44.234539 13189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:44.236585 13189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "3e675b880a284b2a89e319ca900a335d"
format_stamp: "Formatted at 2026-08-10 06:20:44 on dist-test-slave-q9h9"
server_key: "032e72cb20a43fbb43fd127d8d56f735"
server_key_iv: "b31e7726b3d6b7a734df81d615045481"
server_key_version: "encryptionkey@0"
I20260810 06:20:44.237113 13189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "3e675b880a284b2a89e319ca900a335d"
format_stamp: "Formatted at 2026-08-10 06:20:44 on dist-test-slave-q9h9"
server_key: "032e72cb20a43fbb43fd127d8d56f735"
server_key_iv: "b31e7726b3d6b7a734df81d615045481"
server_key_version: "encryptionkey@0"
I20260810 06:20:44.240551 13189 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 06:20:44.242928 13208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:44.244015 13189 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:44.244127 13189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "3e675b880a284b2a89e319ca900a335d"
format_stamp: "Formatted at 2026-08-10 06:20:44 on dist-test-slave-q9h9"
server_key: "032e72cb20a43fbb43fd127d8d56f735"
server_key_iv: "b31e7726b3d6b7a734df81d615045481"
server_key_version: "encryptionkey@0"
I20260810 06:20:44.244233 13189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:44.295367 13189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:44.298539 13189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:44.298722 13189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:44.300173 13189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:44.300221 13189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:44.300263 13189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:44.300294 13189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 06:20:44 dist-test-slave-q9h9 krb5kdc[12541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342844, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM
I20260810 06:20:44.310240 13189 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:42951
I20260810 06:20:44.310496 13393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:42951 every 8 connection(s)
I20260810 06:20:44.311331 13189 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:20:44.312495 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 13189
I20260810 06:20:44.312621 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:20:44.312893 32563 external_mini_cluster.cc:1428] Setting key 290458e10a8e159169d73857a77cdd1f
I20260810 06:20:44.313138 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.300943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:56123 (local address 127.31.204.254:43157)
0810 06:20:44.301103 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:44.301109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:44.301948 (+ 839us) server_negotiation.cc:407] Connection header received
0810 06:20:44.303244 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:44.303247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:44.303314 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:44.303389 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:44.304532 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.305046 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.306311 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.306526 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.306614 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.309414 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:44.309417 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:44.309419 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:44.309443 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:44.311006 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:44.311511 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:44.311515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:44.311517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:44.311628 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:44.312027 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:44.312030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:44.312031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:44.312269 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:44.312358 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:44.312859 (+ 501us) server_negotiation.cc:299] Negotiation successful
0810 06:20:44.312980 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 06:20:44.313877 13222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.301315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:43157 (local address 127.31.204.195:56123)
0810 06:20:44.301785 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:44.301836 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:44.303036 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:44.303565 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:44.303571 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:44.303882 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:44.304395 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.304415 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.305166 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.305169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.306193 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.306203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.306679 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.306683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.306843 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.307358 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:44.307374 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:44.309252 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:44.311114 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:44.311119 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:44.311129 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:44.311366 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:44.311745 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:44.311748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:44.311750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:44.311913 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:44.312371 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:44.312377 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:44.312613 (+ 236us) client_negotiation.cc:769] Sending connection context
0810 06:20:44.312805 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 06:20:44.313091 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
I20260810 06:20:44.315088 13394 heartbeater.cc:347] Connected to a master server at 127.31.204.254:43157
I20260810 06:20:44.315234 13394 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:44.315709 13394 heartbeater.cc:511] Master 127.31.204.254:43157 requested a full tablet report, sending...
I20260810 06:20:44.316705 12615 ts_manager.cc:194] Registered new tserver with Master: 3e675b880a284b2a89e319ca900a335d (127.31.204.195:42951)
I20260810 06:20:44.317392 12615 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:56123
I20260810 06:20:44.323348 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.317209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:51407 (local address 127.31.204.254:43157)
0810 06:20:44.317319 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:44.317323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:44.317544 (+ 221us) server_negotiation.cc:407] Connection header received
0810 06:20:44.317661 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:44.317664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:44.317711 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:44.317777 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:44.318235 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.318712 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.319474 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.319688 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.319769 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.320697 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:44.320700 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:44.320703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:44.320727 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:44.321903 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:44.322308 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:44.322311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:44.322313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:44.322380 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:44.322619 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:44.322622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:44.322624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:44.322796 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:44.322875 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:44.323116 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 06:20:44.323212 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:20:44.323348 13404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.317118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:43157 (local address 127.31.204.195:51407)
0810 06:20:44.317470 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:44.317484 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:44.317573 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:44.317816 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:44.317819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:44.317995 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:44.318133 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.318139 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.318831 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.318835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.319370 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.319378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.319798 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.319802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.319892 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.320288 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:44.320301 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:44.320571 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:44.322003 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:44.322007 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:44.322009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:44.322216 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:44.322460 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:44.322463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:44.322465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:44.322540 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:44.322885 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:44.322888 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:44.322970 (+ 82us) client_negotiation.cc:769] Sending connection context
0810 06:20:44.323051 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 06:20:44.323181 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":95,"threads_started":1}
I20260810 06:20:44.327263 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 06:20:44 dist-test-slave-q9h9 krb5kdc[12541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342844, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 06:20:44 dist-test-slave-q9h9 krb5kdc[12541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:44 dist-test-slave-q9h9 krb5kdc[12541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 06:20:44.349725 12693 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:44.344173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.31.204.254:43157)
0810 06:20:44.344283 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:44.344286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:44.344298 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:20:44.344330 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:44.344332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:44.344368 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:44.344432 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:44.345024 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.345532 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.346358 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.346557 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.346638 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.349609 (+ 2971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44218: BlockingRecv error: recv got EOF from 127.0.0.1:44218 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 10 06:20:44 dist-test-slave-q9h9 krb5kdc[12541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342844, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM
I20260810 06:20:44.358563 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.351077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44220 (local address 127.31.204.254:43157)
0810 06:20:44.351188 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:44.351192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:44.351335 (+ 143us) server_negotiation.cc:407] Connection header received
0810 06:20:44.351431 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:44.351435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:44.351473 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:44.351536 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:44.352095 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.352565 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.353302 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.353517 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.353598 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.355789 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:44.355792 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:44.355794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:44.355815 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:44.357025 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:44.357459 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:44.357461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:44.357463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:44.357525 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:44.357791 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:44.357794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:44.357796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:44.357947 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:44.358082 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:44.358325 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 06:20:44.358422 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 06:20:44.361147 12615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:20:44.364115 12615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:20:44.377125 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.371994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33642 (local address 127.31.204.193:42859)
0810 06:20:44.372625 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:44.372631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:44.372651 (+ 20us) server_negotiation.cc:407] Connection header received
0810 06:20:44.372718 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:44.372723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:44.372926 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:44.373068 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:44.373336 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.374432 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.375854 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.376503 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.376624 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.376761 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:44.376912 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 06:20:44.377002 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":73,"threads_started":1}
I20260810 06:20:44.377128 13429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.371857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:42859 (local address 127.0.0.1:33642)
0810 06:20:44.372163 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:44.372207 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:44.372321 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:44.373049 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:44.373054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:44.373076 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:44.373202 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.373209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.374543 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.374546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.375720 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.375732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.376623 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.376627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.376733 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.376768 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 06:20:44.376871 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 06:20:44.376988 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":78,"threads_started":1}
I20260810 06:20:44.377911 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.372580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:42951 (local address 127.0.0.1:40242)
0810 06:20:44.373291 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:44.373303 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:44.373377 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:44.373848 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:44.373851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:44.373868 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:44.373949 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.373955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.375251 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.375254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.376502 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.376513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.377550 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.377554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.377639 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.377661 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:20:44.377719 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 06:20:44.377785 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":67,"thread_start_us":94,"threads_started":1}
I20260810 06:20:44.378094 13433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.371966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:46381 (local address 127.0.0.1:58252)
0810 06:20:44.372824 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:44.372837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:44.372918 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:44.374041 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:44.374044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:44.374062 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:44.374135 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.374142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.375175 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.375179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.376504 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.376513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.377734 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.377738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.377824 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.377844 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 06:20:44.377905 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 06:20:44.377965 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":75,"thread_start_us":106,"threads_started":1}
I20260810 06:20:44.378541 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.372622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58252 (local address 127.31.204.194:46381)
0810 06:20:44.373618 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:44.373629 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:44.373649 (+ 20us) server_negotiation.cc:407] Connection header received
0810 06:20:44.373716 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:44.373721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:44.373887 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:44.374041 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:44.374242 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.375065 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.376641 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.377417 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.377549 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.377668 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:44.378332 (+ 664us) server_negotiation.cc:299] Negotiation successful
0810 06:20:44.378412 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":902,"thread_start_us":101,"threads_started":1}
I20260810 06:20:44.378545 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.373071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40242 (local address 127.31.204.195:42951)
0810 06:20:44.373481 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:44.373496 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:44.373517 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:20:44.373577 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:44.373581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:44.373722 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:44.373845 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:44.374094 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.375093 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.376640 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.377417 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.378135 (+ 718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.378273 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:44.378323 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 06:20:44.378405 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":114,"threads_started":1}
I20260810 06:20:44.380404 12838 tablet_service.cc:1467] Processing CreateTablet for tablet 0dbdb1269a8a4c6eba18e3a909f8fc6b (DEFAULT_TABLE table=test-table [id=6ae4dd1e735344bba89e4118d27ff1c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:44.381328 12838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dbdb1269a8a4c6eba18e3a909f8fc6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:44.380951 13067 tablet_service.cc:1467] Processing CreateTablet for tablet 0dbdb1269a8a4c6eba18e3a909f8fc6b (DEFAULT_TABLE table=test-table [id=6ae4dd1e735344bba89e4118d27ff1c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:44.381953 13067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dbdb1269a8a4c6eba18e3a909f8fc6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:44.381578 13283 tablet_service.cc:1467] Processing CreateTablet for tablet 0dbdb1269a8a4c6eba18e3a909f8fc6b (DEFAULT_TABLE table=test-table [id=6ae4dd1e735344bba89e4118d27ff1c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:44.382578 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dbdb1269a8a4c6eba18e3a909f8fc6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:44.387252 13437 tablet_bootstrap.cc:492] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5: Bootstrap starting.
I20260810 06:20:44.389192 13438 tablet_bootstrap.cc:492] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13: Bootstrap starting.
I20260810 06:20:44.389959 13437 tablet_bootstrap.cc:654] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:44.391011 13437 log.cc:824] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:44.391309 13438 tablet_bootstrap.cc:654] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:44.391708 13440 tablet_bootstrap.cc:492] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d: Bootstrap starting.
I20260810 06:20:44.392151 13438 log.cc:824] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:44.392820 13437 tablet_bootstrap.cc:492] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5: No bootstrap required, opened a new log
I20260810 06:20:44.393014 13437 ts_tablet_manager.cc:1397] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260810 06:20:44.393642 13440 tablet_bootstrap.cc:654] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:44.394170 13438 tablet_bootstrap.cc:492] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13: No bootstrap required, opened a new log
I20260810 06:20:44.394356 13438 ts_tablet_manager.cc:1397] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260810 06:20:44.394433 13440 log.cc:824] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:44.396250 13440 tablet_bootstrap.cc:492] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d: No bootstrap required, opened a new log
I20260810 06:20:44.396425 13440 ts_tablet_manager.cc:1397] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 06:20:44.396772 13437 raft_consensus.cc:348] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:44.396976 13437 raft_consensus.cc:374] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:44.397029 13437 raft_consensus.cc:729] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da22030ce30d4474b5a0043fccea9bf5, State: Initialized, Role: FOLLOWER
I20260810 06:20:44.397408 13437 consensus_queue.cc:260] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:44.397943 12951 heartbeater.cc:503] Master 127.31.204.254:43157 was elected leader, sending a full tablet report...
I20260810 06:20:44.398078 13438 raft_consensus.cc:348] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:44.398339 13438 raft_consensus.cc:374] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:44.398399 13438 raft_consensus.cc:729] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593684227d124bdda7254e54ed161e13, State: Initialized, Role: FOLLOWER
I20260810 06:20:44.398788 13438 consensus_queue.cc:260] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:44.399281 13437 ts_tablet_manager.cc:1428] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260810 06:20:44.399691 13438 ts_tablet_manager.cc:1428] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 06:20:44.400055 13171 heartbeater.cc:503] Master 127.31.204.254:43157 was elected leader, sending a full tablet report...
I20260810 06:20:44.400040 13440 raft_consensus.cc:348] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:44.400233 13440 raft_consensus.cc:374] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:44.400269 13440 raft_consensus.cc:729] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e675b880a284b2a89e319ca900a335d, State: Initialized, Role: FOLLOWER
I20260810 06:20:44.400565 13440 consensus_queue.cc:260] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:44.401090 13394 heartbeater.cc:503] Master 127.31.204.254:43157 was elected leader, sending a full tablet report...
I20260810 06:20:44.401376 13440 ts_tablet_manager.cc:1428] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 06:20:44.472779 13446 raft_consensus.cc:482] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:20:44.473039 13446 raft_consensus.cc:504] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:44.474370 13446 leader_election.cc:288] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da22030ce30d4474b5a0043fccea9bf5 (127.31.204.193:42859), 3e675b880a284b2a89e319ca900a335d (127.31.204.195:42951)
I20260810 06:20:44.478041 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.474734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:33467 (local address 127.31.204.195:42951)
0810 06:20:44.474886 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:44.474891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:44.475110 (+ 219us) server_negotiation.cc:407] Connection header received
0810 06:20:44.475221 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:44.475224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:44.475267 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:44.475328 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:44.475577 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.476072 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.477135 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.477513 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.477615 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.477671 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:44.477840 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 06:20:44.477906 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 06:20:44.478056 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.474610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:42951 (local address 127.31.204.194:33467)
0810 06:20:44.475017 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:44.475027 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:44.475124 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:44.475350 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:44.475353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:44.475368 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:44.475477 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.475482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.476177 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.476180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.477015 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.477024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.477621 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.477624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.477726 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.477746 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 06:20:44.477813 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 06:20:44.477880 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":22,"thread_start_us":118,"threads_started":1}
I20260810 06:20:44.478271 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.474573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:42859 (local address 127.31.204.194:55137)
0810 06:20:44.474748 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:44.474764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:44.474871 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:44.475126 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:44.475129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:44.475147 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:44.475255 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.475260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.475973 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.475976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.477205 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.477218 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.477857 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.477861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.477960 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.477977 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 06:20:44.478076 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 06:20:44.478151 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":21}
I20260810 06:20:44.478276 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.474701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:55137 (local address 127.31.204.193:42859)
0810 06:20:44.474836 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:44.474841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:44.474857 (+ 16us) server_negotiation.cc:407] Connection header received
0810 06:20:44.474984 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:44.474987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:44.475034 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:44.475095 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:44.475363 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.475866 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.477338 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.477747 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.477851 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.477901 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:44.478099 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 06:20:44.478170 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:20:44.478616 13323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbdb1269a8a4c6eba18e3a909f8fc6b" candidate_uuid: "593684227d124bdda7254e54ed161e13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e675b880a284b2a89e319ca900a335d" is_pre_election: true
I20260810 06:20:44.478883 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbdb1269a8a4c6eba18e3a909f8fc6b" candidate_uuid: "593684227d124bdda7254e54ed161e13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da22030ce30d4474b5a0043fccea9bf5" is_pre_election: true
I20260810 06:20:44.478940 13323 raft_consensus.cc:2454] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 593684227d124bdda7254e54ed161e13 in term 0.
I20260810 06:20:44.479142 12873 raft_consensus.cc:2454] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 593684227d124bdda7254e54ed161e13 in term 0.
I20260810 06:20:44.479425 12991 leader_election.cc:302] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e675b880a284b2a89e319ca900a335d, 593684227d124bdda7254e54ed161e13; no voters:
I20260810 06:20:44.479717 13446 raft_consensus.cc:2781] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:20:44.479803 13446 raft_consensus.cc:482] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:20:44.479852 13446 raft_consensus.cc:3037] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:44.480866 13446 raft_consensus.cc:504] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:44.481156 13446 leader_election.cc:288] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [CANDIDATE]: Term 1 election: Requested vote from peers da22030ce30d4474b5a0043fccea9bf5 (127.31.204.193:42859), 3e675b880a284b2a89e319ca900a335d (127.31.204.195:42951)
I20260810 06:20:44.481485 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbdb1269a8a4c6eba18e3a909f8fc6b" candidate_uuid: "593684227d124bdda7254e54ed161e13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da22030ce30d4474b5a0043fccea9bf5"
I20260810 06:20:44.481504 13323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbdb1269a8a4c6eba18e3a909f8fc6b" candidate_uuid: "593684227d124bdda7254e54ed161e13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e675b880a284b2a89e319ca900a335d"
I20260810 06:20:44.481601 12873 raft_consensus.cc:3037] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:44.481601 13323 raft_consensus.cc:3037] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:44.482893 12873 raft_consensus.cc:2454] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593684227d124bdda7254e54ed161e13 in term 1.
I20260810 06:20:44.482893 13323 raft_consensus.cc:2454] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593684227d124bdda7254e54ed161e13 in term 1.
I20260810 06:20:44.483208 12991 leader_election.cc:302] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e675b880a284b2a89e319ca900a335d, 593684227d124bdda7254e54ed161e13; no voters:
I20260810 06:20:44.483419 13446 raft_consensus.cc:2781] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:44.483620 13446 raft_consensus.cc:686] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [term 1 LEADER]: Becoming Leader. State: Replica: 593684227d124bdda7254e54ed161e13, State: Running, Role: LEADER
I20260810 06:20:44.483938 13446 consensus_queue.cc:237] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:44.486881 12615 catalog_manager.cc:5495] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 593684227d124bdda7254e54ed161e13 (127.31.204.194). New cstate: current_term: 1 leader_uuid: "593684227d124bdda7254e54ed161e13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } health_report { overall_health: UNKNOWN } } }
I20260810 06:20:44.504897 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.501464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.31.204.194:46381)
0810 06:20:44.501588 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:44.501593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:44.501606 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:20:44.501775 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:44.501778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:44.501829 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:44.501912 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:44.502186 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.502722 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.503561 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.503778 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.503861 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.504121 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:44.504518 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:44.504618 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:44.504693 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 06:20:44.504745 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:20:44.510658 12873 raft_consensus.cc:1261] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5 [term 1 FOLLOWER]: Refusing update from remote peer 593684227d124bdda7254e54ed161e13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:44.510958 13323 raft_consensus.cc:1261] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d [term 1 FOLLOWER]: Refusing update from remote peer 593684227d124bdda7254e54ed161e13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:44.511369 13450 consensus_queue.cc:1035] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:44.511579 13446 consensus_queue.cc:1035] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:44.527094 12615 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:44.527251 12615 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:44.529363 12615 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c: Sending DeleteTablet for 3 replicas of tablet 0dbdb1269a8a4c6eba18e3a909f8fc6b
I20260810 06:20:44.530078 13067 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dbdb1269a8a4c6eba18e3a909f8fc6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:44 UTC) from {username='oryx', principal='oryx/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:58252
I20260810 06:20:44.530197 13283 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dbdb1269a8a4c6eba18e3a909f8fc6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:44 UTC) from {username='oryx', principal='oryx/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:40242
I20260810 06:20:44.530562 13477 tablet_replica.cc:330] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13: stopping tablet replica
I20260810 06:20:44.530579 12838 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dbdb1269a8a4c6eba18e3a909f8fc6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:44 UTC) from {username='oryx', principal='oryx/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:33642
I20260810 06:20:44.530858 13477 raft_consensus.cc:2229] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:20:44.531158 13477 raft_consensus.cc:2258] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:44.531162 13479 tablet_replica.cc:330] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d: stopping tablet replica
I20260810 06:20:44.531226 12615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:20:44.531611 12615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:20:44.531745 13479 raft_consensus.cc:2229] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:44.532087 13479 raft_consensus.cc:2258] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:44.532591 13477 ts_tablet_manager.cc:1905] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:44.532871 13480 tablet_replica.cc:330] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5: stopping tablet replica
I20260810 06:20:44.533142 13480 raft_consensus.cc:2229] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:44.533423 13480 raft_consensus.cc:2258] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:44.534794 13480 ts_tablet_manager.cc:1905] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:44.535702 13477 ts_tablet_manager.cc:1918] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:20:44.535794 13477 log.cc:1196] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/0dbdb1269a8a4c6eba18e3a909f8fc6b
I20260810 06:20:44.536146 13477 ts_tablet_manager.cc:1939] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 593684227d124bdda7254e54ed161e13: Deleting consensus metadata
I20260810 06:20:44.537071 12838 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf9a137bcf845acb4a567ad51419190 (DEFAULT_TABLE table=test-table [id=aca064247744465884749147d68d2b18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:44.537164 12593 catalog_manager.cc:4841] TS 593684227d124bdda7254e54ed161e13 (127.31.204.194:46381): tablet 0dbdb1269a8a4c6eba18e3a909f8fc6b (table test-table [id=6ae4dd1e735344bba89e4118d27ff1c4]) successfully deleted
I20260810 06:20:44.537315 12838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf9a137bcf845acb4a567ad51419190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:44.538256 13067 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf9a137bcf845acb4a567ad51419190 (DEFAULT_TABLE table=test-table [id=aca064247744465884749147d68d2b18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:44.538475 13067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf9a137bcf845acb4a567ad51419190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:44.538518 13480 ts_tablet_manager.cc:1918] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:20:44.538601 13480 log.cc:1196] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/wal/wals/0dbdb1269a8a4c6eba18e3a909f8fc6b
I20260810 06:20:44.538882 13480 ts_tablet_manager.cc:1939] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P da22030ce30d4474b5a0043fccea9bf5: Deleting consensus metadata
I20260810 06:20:44.539556 13437 tablet_bootstrap.cc:492] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5: Bootstrap starting.
I20260810 06:20:44.540345 13437 tablet_bootstrap.cc:654] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:44.540503 13438 tablet_bootstrap.cc:492] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13: Bootstrap starting.
I20260810 06:20:44.541126 13283 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf9a137bcf845acb4a567ad51419190 (DEFAULT_TABLE table=test-table [id=aca064247744465884749147d68d2b18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:44.541199 12597 catalog_manager.cc:4841] TS da22030ce30d4474b5a0043fccea9bf5 (127.31.204.193:42859): tablet 0dbdb1269a8a4c6eba18e3a909f8fc6b (table test-table [id=6ae4dd1e735344bba89e4118d27ff1c4]) successfully deleted
I20260810 06:20:44.541373 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf9a137bcf845acb4a567ad51419190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:44.541419 13438 tablet_bootstrap.cc:654] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:44.541965 13437 tablet_bootstrap.cc:492] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5: No bootstrap required, opened a new log
I20260810 06:20:44.542061 13437 ts_tablet_manager.cc:1397] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260810 06:20:44.542789 13437 raft_consensus.cc:348] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:44.542896 13437 raft_consensus.cc:374] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:44.542929 13437 raft_consensus.cc:729] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da22030ce30d4474b5a0043fccea9bf5, State: Initialized, Role: FOLLOWER
I20260810 06:20:44.542989 13438 tablet_bootstrap.cc:492] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13: No bootstrap required, opened a new log
I20260810 06:20:44.543056 13438 ts_tablet_manager.cc:1397] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 06:20:44.543040 13437 consensus_queue.cc:260] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:44.543303 13437 ts_tablet_manager.cc:1428] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:44.543521 13438 raft_consensus.cc:348] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:44.543617 13438 raft_consensus.cc:374] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:44.543648 13438 raft_consensus.cc:729] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593684227d124bdda7254e54ed161e13, State: Initialized, Role: FOLLOWER
I20260810 06:20:44.543756 13438 consensus_queue.cc:260] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:44.544020 13438 ts_tablet_manager.cc:1428] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:44.544241 13440 tablet_bootstrap.cc:492] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d: Bootstrap starting.
I20260810 06:20:44.545233 13440 tablet_bootstrap.cc:654] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:44.545375 13479 ts_tablet_manager.cc:1905] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:44.547678 13440 tablet_bootstrap.cc:492] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d: No bootstrap required, opened a new log
I20260810 06:20:44.547756 13440 ts_tablet_manager.cc:1397] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 06:20:44.547797 13479 ts_tablet_manager.cc:1918] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:20:44.547891 13479 log.cc:1196] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/0dbdb1269a8a4c6eba18e3a909f8fc6b
I20260810 06:20:44.548244 13479 ts_tablet_manager.cc:1939] T 0dbdb1269a8a4c6eba18e3a909f8fc6b P 3e675b880a284b2a89e319ca900a335d: Deleting consensus metadata
I20260810 06:20:44.548223 13440 raft_consensus.cc:348] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:44.548373 13440 raft_consensus.cc:374] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:44.548455 13440 raft_consensus.cc:729] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e675b880a284b2a89e319ca900a335d, State: Initialized, Role: FOLLOWER
I20260810 06:20:44.548578 13440 consensus_queue.cc:260] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:44.548874 13440 ts_tablet_manager.cc:1428] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:44.549389 12598 catalog_manager.cc:4841] TS 3e675b880a284b2a89e319ca900a335d (127.31.204.195:42951): tablet 0dbdb1269a8a4c6eba18e3a909f8fc6b (table test-table [id=6ae4dd1e735344bba89e4118d27ff1c4]) successfully deleted
W20260810 06:20:44.555469 13172 tablet.cc:2369] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:20:44.562213 13396 tablet.cc:2369] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:20:44.583832 12952 tablet.cc:2369] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:20:44.898466 13445 raft_consensus.cc:482] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:20:44.898628 13445 raft_consensus.cc:504] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:44.899672 13445 leader_election.cc:288] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 593684227d124bdda7254e54ed161e13 (127.31.204.194:46381), 3e675b880a284b2a89e319ca900a335d (127.31.204.195:42951)
I20260810 06:20:44.903916 13492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.900016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:42951 (local address 127.31.204.193:53799)
0810 06:20:44.900342 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:44.900356 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:44.900466 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:44.900687 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:44.900690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:44.900707 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:44.900814 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.900820 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.901540 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.901543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.902802 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.902811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.903487 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.903491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.903582 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.903622 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 06:20:44.903690 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 06:20:44.903747 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":124,"thread_start_us":101,"threads_started":1}
I20260810 06:20:44.903913 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.900154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:53799 (local address 127.31.204.195:42951)
0810 06:20:44.900268 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:44.900272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:44.900443 (+ 171us) server_negotiation.cc:407] Connection header received
0810 06:20:44.900545 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:44.900547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:44.900588 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:44.900661 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:44.900921 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.901392 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.902923 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.903376 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.903468 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.903508 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:44.903722 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 06:20:44.903793 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:20:44.903960 13490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.899923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:46381 (local address 127.31.204.193:53889)
0810 06:20:44.900246 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:44.900261 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:44.900375 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:44.900628 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:44.900631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:44.900647 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:20:44.900757 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.900763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.901482 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.901485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.902775 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:44.902785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.903585 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.903588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:44.903684 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.903708 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 06:20:44.903778 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 06:20:44.903837 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":49,"thread_start_us":75,"threads_started":1}
I20260810 06:20:44.904002 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:44.900015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:53889 (local address 127.31.204.194:46381)
0810 06:20:44.900157 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:44.900161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:44.900348 (+ 187us) server_negotiation.cc:407] Connection header received
0810 06:20:44.900474 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:44.900477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:44.900529 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:44.900594 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:20:44.900864 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.901374 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.902889 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:44.903465 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:44.903580 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:44.903631 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:44.903813 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 06:20:44.903897 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 06:20:44.904439 13095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf9a137bcf845acb4a567ad51419190" candidate_uuid: "da22030ce30d4474b5a0043fccea9bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593684227d124bdda7254e54ed161e13" is_pre_election: true
I20260810 06:20:44.904646 13095 raft_consensus.cc:2454] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da22030ce30d4474b5a0043fccea9bf5 in term 0.
I20260810 06:20:44.904685 13323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf9a137bcf845acb4a567ad51419190" candidate_uuid: "da22030ce30d4474b5a0043fccea9bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e675b880a284b2a89e319ca900a335d" is_pre_election: true
I20260810 06:20:44.904814 13323 raft_consensus.cc:2454] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da22030ce30d4474b5a0043fccea9bf5 in term 0.
I20260810 06:20:44.905055 12755 leader_election.cc:302] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593684227d124bdda7254e54ed161e13, da22030ce30d4474b5a0043fccea9bf5; no voters:
I20260810 06:20:44.905311 13445 raft_consensus.cc:2781] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:20:44.905365 13445 raft_consensus.cc:482] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:20:44.905403 13445 raft_consensus.cc:3037] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:44.906390 13445 raft_consensus.cc:504] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:44.906702 13445 leader_election.cc:288] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [CANDIDATE]: Term 1 election: Requested vote from peers 593684227d124bdda7254e54ed161e13 (127.31.204.194:46381), 3e675b880a284b2a89e319ca900a335d (127.31.204.195:42951)
I20260810 06:20:44.906992 13095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf9a137bcf845acb4a567ad51419190" candidate_uuid: "da22030ce30d4474b5a0043fccea9bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593684227d124bdda7254e54ed161e13"
I20260810 06:20:44.907014 13323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf9a137bcf845acb4a567ad51419190" candidate_uuid: "da22030ce30d4474b5a0043fccea9bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e675b880a284b2a89e319ca900a335d"
I20260810 06:20:44.907115 13323 raft_consensus.cc:3037] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:44.907114 13095 raft_consensus.cc:3037] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:44.907943 13323 raft_consensus.cc:2454] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da22030ce30d4474b5a0043fccea9bf5 in term 1.
I20260810 06:20:44.908032 13095 raft_consensus.cc:2454] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da22030ce30d4474b5a0043fccea9bf5 in term 1.
I20260810 06:20:44.908201 12759 leader_election.cc:302] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e675b880a284b2a89e319ca900a335d, da22030ce30d4474b5a0043fccea9bf5; no voters:
I20260810 06:20:44.908353 13445 raft_consensus.cc:2781] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:44.908547 13445 raft_consensus.cc:686] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [term 1 LEADER]: Becoming Leader. State: Replica: da22030ce30d4474b5a0043fccea9bf5, State: Running, Role: LEADER
I20260810 06:20:44.908776 13445 consensus_queue.cc:237] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:44.910830 12617 catalog_manager.cc:5495] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to da22030ce30d4474b5a0043fccea9bf5 (127.31.204.193). New cstate: current_term: 1 leader_uuid: "da22030ce30d4474b5a0043fccea9bf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } health_report { overall_health: UNKNOWN } } }
Aug 10 06:20:45 dist-test-slave-q9h9 krb5kdc[12541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342843, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.254@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM
I20260810 06:20:45.027973 13511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:45.019703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:43157 (local address 127.0.0.1:44234)
0810 06:20:45.020004 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:45.020018 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:45.020134 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:45.020365 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:45.020368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:45.020771 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:45.020868 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:45.020875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.021759 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.021762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:45.022551 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:45.022561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.022989 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.022992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:45.023086 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:45.023633 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:45.023645 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:45.025007 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:45.026631 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:45.026640 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:45.026644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:45.026873 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:45.027084 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:45.027086 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:45.027087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:45.027156 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:45.027470 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:45.027475 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:45.027581 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 06:20:45.027667 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 06:20:45.027793 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":2,"thread_start_us":100,"threads_started":1}
I20260810 06:20:45.027982 13512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:45.019779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.31.204.254:43157)
0810 06:20:45.019992 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:45.020000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:45.020088 (+ 88us) server_negotiation.cc:407] Connection header received
0810 06:20:45.020212 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:45.020216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:45.020258 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:45.020329 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:45.020981 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.021651 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.022663 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.022892 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.022979 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:45.025156 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:45.025159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:45.025161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:45.025184 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:45.026544 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:45.026947 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:45.026949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:45.026950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:45.027022 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:45.027234 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:45.027237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:45.027238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:45.027394 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:45.027463 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:45.027721 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 06:20:45.027847 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":165,"thread_start_us":81,"threads_started":1}
I20260810 06:20:45.031513 12617 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:44234:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 06:20:45.032193 12617 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:20:45.036650 12838 tablet_service.cc:1467] Processing CreateTablet for tablet a8a56fe3a8754e2596decdbfad0222d3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1afda2b13a904304927a365cbdad9685]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 06:20:45.036744 13067 tablet_service.cc:1467] Processing CreateTablet for tablet a8a56fe3a8754e2596decdbfad0222d3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1afda2b13a904304927a365cbdad9685]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 06:20:45.036801 13283 tablet_service.cc:1467] Processing CreateTablet for tablet a8a56fe3a8754e2596decdbfad0222d3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1afda2b13a904304927a365cbdad9685]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 06:20:45.037060 12838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a56fe3a8754e2596decdbfad0222d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:45.037096 13067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a56fe3a8754e2596decdbfad0222d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:45.037113 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a56fe3a8754e2596decdbfad0222d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:45.039352 13438 tablet_bootstrap.cc:492] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13: Bootstrap starting.
I20260810 06:20:45.039906 13437 tablet_bootstrap.cc:492] T a8a56fe3a8754e2596decdbfad0222d3 P da22030ce30d4474b5a0043fccea9bf5: Bootstrap starting.
I20260810 06:20:45.040194 13438 tablet_bootstrap.cc:654] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:45.040781 13437 tablet_bootstrap.cc:654] T a8a56fe3a8754e2596decdbfad0222d3 P da22030ce30d4474b5a0043fccea9bf5: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:45.041195 13438 tablet_bootstrap.cc:492] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13: No bootstrap required, opened a new log
I20260810 06:20:45.041270 13438 ts_tablet_manager.cc:1397] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:45.041337 13440 tablet_bootstrap.cc:492] T a8a56fe3a8754e2596decdbfad0222d3 P 3e675b880a284b2a89e319ca900a335d: Bootstrap starting.
I20260810 06:20:45.041653 13438 raft_consensus.cc:348] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.041771 13438 raft_consensus.cc:374] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:45.041803 13438 raft_consensus.cc:729] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593684227d124bdda7254e54ed161e13, State: Initialized, Role: FOLLOWER
I20260810 06:20:45.041967 13437 tablet_bootstrap.cc:492] T a8a56fe3a8754e2596decdbfad0222d3 P da22030ce30d4474b5a0043fccea9bf5: No bootstrap required, opened a new log
I20260810 06:20:45.041925 13438 consensus_queue.cc:260] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.042073 13437 ts_tablet_manager.cc:1397] T a8a56fe3a8754e2596decdbfad0222d3 P da22030ce30d4474b5a0043fccea9bf5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:45.042238 13440 tablet_bootstrap.cc:654] T a8a56fe3a8754e2596decdbfad0222d3 P 3e675b880a284b2a89e319ca900a335d: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:45.042212 13438 ts_tablet_manager.cc:1428] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:45.042483 13437 raft_consensus.cc:348] T a8a56fe3a8754e2596decdbfad0222d3 P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.042590 13437 raft_consensus.cc:374] T a8a56fe3a8754e2596decdbfad0222d3 P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:45.042630 13437 raft_consensus.cc:729] T a8a56fe3a8754e2596decdbfad0222d3 P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da22030ce30d4474b5a0043fccea9bf5, State: Initialized, Role: FOLLOWER
I20260810 06:20:45.042734 13437 consensus_queue.cc:260] T a8a56fe3a8754e2596decdbfad0222d3 P da22030ce30d4474b5a0043fccea9bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.042987 13437 ts_tablet_manager.cc:1428] T a8a56fe3a8754e2596decdbfad0222d3 P da22030ce30d4474b5a0043fccea9bf5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:45.042935 13450 tablet_replica.cc:439] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } } }
I20260810 06:20:45.043063 13450 tablet_replica.cc:442] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:45.043161 13445 tablet_replica.cc:439] T a8a56fe3a8754e2596decdbfad0222d3 P da22030ce30d4474b5a0043fccea9bf5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } } }
I20260810 06:20:45.043286 13445 tablet_replica.cc:442] T a8a56fe3a8754e2596decdbfad0222d3 P da22030ce30d4474b5a0043fccea9bf5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:45.043419 13440 tablet_bootstrap.cc:492] T a8a56fe3a8754e2596decdbfad0222d3 P 3e675b880a284b2a89e319ca900a335d: No bootstrap required, opened a new log
I20260810 06:20:45.043505 13440 ts_tablet_manager.cc:1397] T a8a56fe3a8754e2596decdbfad0222d3 P 3e675b880a284b2a89e319ca900a335d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:45.043849 13440 raft_consensus.cc:348] T a8a56fe3a8754e2596decdbfad0222d3 P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.043936 13440 raft_consensus.cc:374] T a8a56fe3a8754e2596decdbfad0222d3 P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:45.043956 13440 raft_consensus.cc:729] T a8a56fe3a8754e2596decdbfad0222d3 P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e675b880a284b2a89e319ca900a335d, State: Initialized, Role: FOLLOWER
I20260810 06:20:45.044051 13440 consensus_queue.cc:260] T a8a56fe3a8754e2596decdbfad0222d3 P 3e675b880a284b2a89e319ca900a335d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.044312 13440 ts_tablet_manager.cc:1428] T a8a56fe3a8754e2596decdbfad0222d3 P 3e675b880a284b2a89e319ca900a335d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:45.044816 13448 tablet_replica.cc:439] T a8a56fe3a8754e2596decdbfad0222d3 P 3e675b880a284b2a89e319ca900a335d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } } }
I20260810 06:20:45.044948 13448 tablet_replica.cc:442] T a8a56fe3a8754e2596decdbfad0222d3 P 3e675b880a284b2a89e319ca900a335d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:45.150426 13450 raft_consensus.cc:482] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:20:45.150553 13450 raft_consensus.cc:504] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.150985 13450 leader_election.cc:288] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da22030ce30d4474b5a0043fccea9bf5 (127.31.204.193:42859), 3e675b880a284b2a89e319ca900a335d (127.31.204.195:42951)
I20260810 06:20:45.151417 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a56fe3a8754e2596decdbfad0222d3" candidate_uuid: "593684227d124bdda7254e54ed161e13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da22030ce30d4474b5a0043fccea9bf5" is_pre_election: true
I20260810 06:20:45.151415 13323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a56fe3a8754e2596decdbfad0222d3" candidate_uuid: "593684227d124bdda7254e54ed161e13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e675b880a284b2a89e319ca900a335d" is_pre_election: true
I20260810 06:20:45.151561 12873 raft_consensus.cc:2454] T a8a56fe3a8754e2596decdbfad0222d3 P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 593684227d124bdda7254e54ed161e13 in term 0.
I20260810 06:20:45.151575 13323 raft_consensus.cc:2454] T a8a56fe3a8754e2596decdbfad0222d3 P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 593684227d124bdda7254e54ed161e13 in term 0.
I20260810 06:20:45.151839 12990 leader_election.cc:302] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593684227d124bdda7254e54ed161e13, da22030ce30d4474b5a0043fccea9bf5; no voters:
I20260810 06:20:45.152029 13450 raft_consensus.cc:2781] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:20:45.152091 13450 raft_consensus.cc:482] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:20:45.152128 13450 raft_consensus.cc:3037] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:45.153116 13450 raft_consensus.cc:504] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.153414 13450 leader_election.cc:288] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 [CANDIDATE]: Term 1 election: Requested vote from peers da22030ce30d4474b5a0043fccea9bf5 (127.31.204.193:42859), 3e675b880a284b2a89e319ca900a335d (127.31.204.195:42951)
I20260810 06:20:45.153724 13323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a56fe3a8754e2596decdbfad0222d3" candidate_uuid: "593684227d124bdda7254e54ed161e13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e675b880a284b2a89e319ca900a335d"
I20260810 06:20:45.153815 13323 raft_consensus.cc:3037] T a8a56fe3a8754e2596decdbfad0222d3 P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:45.153757 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a56fe3a8754e2596decdbfad0222d3" candidate_uuid: "593684227d124bdda7254e54ed161e13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da22030ce30d4474b5a0043fccea9bf5"
I20260810 06:20:45.153889 12873 raft_consensus.cc:3037] T a8a56fe3a8754e2596decdbfad0222d3 P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:45.154650 13323 raft_consensus.cc:2454] T a8a56fe3a8754e2596decdbfad0222d3 P 3e675b880a284b2a89e319ca900a335d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593684227d124bdda7254e54ed161e13 in term 1.
I20260810 06:20:45.154965 12991 leader_election.cc:302] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e675b880a284b2a89e319ca900a335d, 593684227d124bdda7254e54ed161e13; no voters:
I20260810 06:20:45.155090 12873 raft_consensus.cc:2454] T a8a56fe3a8754e2596decdbfad0222d3 P da22030ce30d4474b5a0043fccea9bf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593684227d124bdda7254e54ed161e13 in term 1.
I20260810 06:20:45.155131 13450 raft_consensus.cc:2781] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:45.155328 13450 raft_consensus.cc:686] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 [term 1 LEADER]: Becoming Leader. State: Replica: 593684227d124bdda7254e54ed161e13, State: Running, Role: LEADER
I20260810 06:20:45.155476 13450 consensus_queue.cc:237] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.156081 13450 tablet_replica.cc:439] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13: TxnStatusTablet state changed. Reason: New leader 593684227d124bdda7254e54ed161e13. Latest consensus state: current_term: 1 leader_uuid: "593684227d124bdda7254e54ed161e13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } } }
I20260810 06:20:45.156455 13450 tablet_replica.cc:442] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13: This TxnStatusTablet replica's current role is: LEADER
I20260810 06:20:45.156749 13523 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 06:20:45.156903 13523 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 06:20:45.157330 12617 catalog_manager.cc:5495] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 593684227d124bdda7254e54ed161e13 (127.31.204.194). New cstate: current_term: 1 leader_uuid: "593684227d124bdda7254e54ed161e13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } health_report { overall_health: UNKNOWN } } }
I20260810 06:20:45.170701 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:45.166669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.31.204.194:46381)
0810 06:20:45.166791 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:45.166796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:45.166877 (+ 81us) server_negotiation.cc:407] Connection header received
0810 06:20:45.167048 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:45.167051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:45.167102 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:45.167183 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:45.167478 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.168277 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.169477 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.169837 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.169943 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:45.170241 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:45.170328 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:45.170406 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:45.170506 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 06:20:45.170563 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:20:45.170708 13511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:45.166585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:46381 (local address 127.0.0.1:58272)
0810 06:20:45.166820 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:45.166834 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:45.166938 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:45.167208 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:45.167211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:45.167219 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:20:45.167327 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:45.167333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.168409 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.168413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:45.169361 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:45.169371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.169943 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.169946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:45.170056 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:45.170109 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:20:45.170414 (+ 305us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:20:45.170424 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 06:20:45.170479 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 06:20:45.170563 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260810 06:20:45.173478 13323 raft_consensus.cc:1261] T a8a56fe3a8754e2596decdbfad0222d3 P 3e675b880a284b2a89e319ca900a335d [term 1 FOLLOWER]: Refusing update from remote peer 593684227d124bdda7254e54ed161e13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:45.173935 12873 raft_consensus.cc:1261] T a8a56fe3a8754e2596decdbfad0222d3 P da22030ce30d4474b5a0043fccea9bf5 [term 1 FOLLOWER]: Refusing update from remote peer 593684227d124bdda7254e54ed161e13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:45.173906 13517 consensus_queue.cc:1035] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:45.174418 13450 consensus_queue.cc:1035] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:45.175710 13448 tablet_replica.cc:439] T a8a56fe3a8754e2596decdbfad0222d3 P 3e675b880a284b2a89e319ca900a335d: TxnStatusTablet state changed. Reason: New leader 593684227d124bdda7254e54ed161e13. Latest consensus state: current_term: 1 leader_uuid: "593684227d124bdda7254e54ed161e13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } } }
I20260810 06:20:45.175827 13448 tablet_replica.cc:442] T a8a56fe3a8754e2596decdbfad0222d3 P 3e675b880a284b2a89e319ca900a335d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:45.175779 13445 tablet_replica.cc:439] T a8a56fe3a8754e2596decdbfad0222d3 P da22030ce30d4474b5a0043fccea9bf5: TxnStatusTablet state changed. Reason: New leader 593684227d124bdda7254e54ed161e13. Latest consensus state: current_term: 1 leader_uuid: "593684227d124bdda7254e54ed161e13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } } }
I20260810 06:20:45.175879 13445 tablet_replica.cc:442] T a8a56fe3a8754e2596decdbfad0222d3 P da22030ce30d4474b5a0043fccea9bf5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:45.180147 13450 tablet_replica.cc:439] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "593684227d124bdda7254e54ed161e13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } } }
I20260810 06:20:45.180256 13450 tablet_replica.cc:442] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13: This TxnStatusTablet replica's current role is: LEADER
I20260810 06:20:45.180351 13450 tablet_replica.cc:439] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "593684227d124bdda7254e54ed161e13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } } }
I20260810 06:20:45.180408 13450 tablet_replica.cc:442] T a8a56fe3a8754e2596decdbfad0222d3 P 593684227d124bdda7254e54ed161e13: This TxnStatusTablet replica's current role is: LEADER
I20260810 06:20:45.180580 13445 tablet_replica.cc:439] T a8a56fe3a8754e2596decdbfad0222d3 P da22030ce30d4474b5a0043fccea9bf5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "593684227d124bdda7254e54ed161e13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } } }
I20260810 06:20:45.180678 13445 tablet_replica.cc:442] T a8a56fe3a8754e2596decdbfad0222d3 P da22030ce30d4474b5a0043fccea9bf5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:45.181790 13448 tablet_replica.cc:439] T a8a56fe3a8754e2596decdbfad0222d3 P 3e675b880a284b2a89e319ca900a335d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "593684227d124bdda7254e54ed161e13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } } }
I20260810 06:20:45.181890 13448 tablet_replica.cc:442] T a8a56fe3a8754e2596decdbfad0222d3 P 3e675b880a284b2a89e319ca900a335d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:20:45.182921 13528 mvcc.cc:205] Tried to move back new op lower bound from 7316860293825720320 to 7316860293758185472. Current Snapshot: MvccSnapshot[applied={T|T < 7316860293825720320}]
I20260810 06:20:45.183799 13529 mvcc.cc:205] Tried to move back new op lower bound from 7316860293825720320 to 7316860293758185472. Current Snapshot: MvccSnapshot[applied={T|T < 7316860293825720320}]
I20260810 06:20:45.184849 13531 mvcc.cc:205] Tried to move back new op lower bound from 7316860293825720320 to 7316860293758185472. Current Snapshot: MvccSnapshot[applied={T|T < 7316860293825720320}]
I20260810 06:20:45.194603 13543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:45.187980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.31.204.193:42859)
0810 06:20:45.190220 (+ 2240us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:45.190224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:45.190237 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:20:45.190300 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:45.190303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:45.190365 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:45.190474 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:45.190846 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.191688 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.192928 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.193305 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.193407 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:45.193689 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:45.194207 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:45.194294 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:45.194392 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 06:20:45.194438 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2176,"thread_start_us":70,"threads_started":1}
I20260810 06:20:45.206351 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:45.202063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:40297 (local address 127.31.204.194:46381)
0810 06:20:45.202184 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:45.202188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:45.202202 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:20:45.202244 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:45.202248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:45.202319 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:45.202394 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:45.203184 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.203954 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.205192 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.205523 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.205631 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:45.205914 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:45.205989 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:45.206090 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:45.206169 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 06:20:45.206223 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 06:20:45.206549 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:45.201750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:46381 (local address 127.31.204.193:40297)
0810 06:20:45.202050 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:45.202064 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:45.202150 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:45.202415 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:45.202418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:45.202426 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:20:45.203061 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:45.203072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.204076 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.204080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:45.205065 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:45.205076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.205636 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.205639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:45.205742 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:45.205790 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:20:45.206094 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:20:45.206101 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 06:20:45.206157 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 06:20:45.206234 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":97,"threads_started":1}
I20260810 06:20:45.213675 13323 raft_consensus.cc:1261] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d [term 1 FOLLOWER]: Refusing update from remote peer da22030ce30d4474b5a0043fccea9bf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:45.213992 13095 raft_consensus.cc:1261] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13 [term 1 FOLLOWER]: Refusing update from remote peer da22030ce30d4474b5a0043fccea9bf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:45.214144 13499 consensus_queue.cc:1035] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:45.214512 13499 consensus_queue.cc:1035] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:45.246927 13543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:45.240433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:54927 (local address 127.31.204.193:42859)
0810 06:20:45.240550 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:45.240555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:45.242758 (+ 2203us) server_negotiation.cc:407] Connection header received
0810 06:20:45.242968 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:45.242971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:45.243023 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:45.243086 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:45.243492 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.244263 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.245561 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.245909 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.246011 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:45.246404 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:45.246495 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:45.246544 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:45.246721 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 06:20:45.246781 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:20:45.247663 13551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:45.240031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:42859 (local address 127.31.204.194:54927)
0810 06:20:45.242667 (+ 2636us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:45.242686 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:45.242842 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:45.243239 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:45.243243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:45.243252 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:20:45.243363 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:45.243370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.244422 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.244425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:45.245395 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:45.245409 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.246069 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.246074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:45.246170 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:45.246228 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:20:45.246613 (+ 385us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:20:45.246620 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 06:20:45.247324 (+ 704us) client_negotiation.cc:240] Negotiation successful
0810 06:20:45.247418 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2497,"thread_start_us":112,"threads_started":1}
I20260810 06:20:45.276983 12616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:45.277156 12616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:45.279616 12616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c: Sending DeleteTablet for 3 replicas of tablet 4cf9a137bcf845acb4a567ad51419190
I20260810 06:20:45.280522 12838 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cf9a137bcf845acb4a567ad51419190 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:45 UTC) from {username='oryx', principal='oryx/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:33642
I20260810 06:20:45.280741 13067 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cf9a137bcf845acb4a567ad51419190 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:45 UTC) from {username='oryx', principal='oryx/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:58252
I20260810 06:20:45.281334 13283 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cf9a137bcf845acb4a567ad51419190 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:45 UTC) from {username='oryx', principal='oryx/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:40242
I20260810 06:20:45.282040 13557 tablet_replica.cc:330] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13: stopping tablet replica
I20260810 06:20:45.282191 13557 raft_consensus.cc:2229] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:45.282306 13557 raft_consensus.cc:2258] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:45.282538 13558 tablet_replica.cc:330] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d: stopping tablet replica
I20260810 06:20:45.282661 13558 raft_consensus.cc:2229] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:45.282760 13558 raft_consensus.cc:2258] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:45.282981 13556 tablet_replica.cc:330] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5: stopping tablet replica
I20260810 06:20:45.283092 13556 raft_consensus.cc:2229] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:20:45.283365 13556 raft_consensus.cc:2258] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:45.284027 13556 ts_tablet_manager.cc:1905] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:45.287098 13558 ts_tablet_manager.cc:1905] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:45.287139 13556 ts_tablet_manager.cc:1918] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:20:45.287235 13556 log.cc:1196] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/wal/wals/4cf9a137bcf845acb4a567ad51419190
I20260810 06:20:45.287510 13556 ts_tablet_manager.cc:1939] T 4cf9a137bcf845acb4a567ad51419190 P da22030ce30d4474b5a0043fccea9bf5: Deleting consensus metadata
I20260810 06:20:45.288203 12597 catalog_manager.cc:4841] TS da22030ce30d4474b5a0043fccea9bf5 (127.31.204.193:42859): tablet 4cf9a137bcf845acb4a567ad51419190 (table test-table [id=aca064247744465884749147d68d2b18]) successfully deleted
I20260810 06:20:45.288775 13557 ts_tablet_manager.cc:1905] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:45.290832 13558 ts_tablet_manager.cc:1918] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:20:45.293103 13558 log.cc:1196] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/4cf9a137bcf845acb4a567ad51419190
I20260810 06:20:45.293443 13558 ts_tablet_manager.cc:1939] T 4cf9a137bcf845acb4a567ad51419190 P 3e675b880a284b2a89e319ca900a335d: Deleting consensus metadata
I20260810 06:20:45.294768 12598 catalog_manager.cc:4841] TS 3e675b880a284b2a89e319ca900a335d (127.31.204.195:42951): tablet 4cf9a137bcf845acb4a567ad51419190 (table test-table [id=aca064247744465884749147d68d2b18]) successfully deleted
I20260810 06:20:45.295382 13557 ts_tablet_manager.cc:1918] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:20:45.295598 13557 log.cc:1196] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/4cf9a137bcf845acb4a567ad51419190
I20260810 06:20:45.295917 13557 ts_tablet_manager.cc:1939] T 4cf9a137bcf845acb4a567ad51419190 P 593684227d124bdda7254e54ed161e13: Deleting consensus metadata
I20260810 06:20:45.296591 12593 catalog_manager.cc:4841] TS 593684227d124bdda7254e54ed161e13 (127.31.204.194:46381): tablet 4cf9a137bcf845acb4a567ad51419190 (table test-table [id=aca064247744465884749147d68d2b18]) successfully deleted
Aug 10 06:20:45 dist-test-slave-q9h9 krb5kdc[12541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 06:20:45 dist-test-slave-q9h9 krb5kdc[12541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:45 dist-test-slave-q9h9 krb5kdc[12541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 06:20:45.328730 13512 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:45.320321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44238 (local address 127.31.204.254:43157)
0810 06:20:45.320453 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:45.320458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:45.321225 (+ 767us) server_negotiation.cc:407] Connection header received
0810 06:20:45.321422 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:45.321426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:45.321473 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:45.321522 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:45.322741 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.323450 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.324803 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.325137 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.325214 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:45.328588 (+ 3374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44238: BlockingRecv error: recv got EOF from 127.0.0.1:44238 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 06:20:45 dist-test-slave-q9h9 krb5kdc[12541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM
I20260810 06:20:45.343809 13512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:45.334448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44246 (local address 127.31.204.254:43157)
0810 06:20:45.334564 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:45.334568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:45.335998 (+ 1430us) server_negotiation.cc:407] Connection header received
0810 06:20:45.336098 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:45.336101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:45.336148 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:45.336230 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:45.336949 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.337420 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.338511 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.338715 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.338782 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:45.341145 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:45.341148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:45.341150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:45.341174 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:45.342436 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:45.342838 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:45.342840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:45.342842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:45.342907 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:45.343143 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:45.343145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:45.343146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:45.343302 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:45.343395 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:45.343594 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 06:20:45.343691 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:20:45.346168 12616 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:20:45.346577 12616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:20:45.351317 12838 tablet_service.cc:1467] Processing CreateTablet for tablet 2d907e58e4a74f5680692eb0632ca8e4 (DEFAULT_TABLE table=test-table [id=8c800c4bc1974b1fb8f9716aa8b9b061]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:45.351353 13067 tablet_service.cc:1467] Processing CreateTablet for tablet 2d907e58e4a74f5680692eb0632ca8e4 (DEFAULT_TABLE table=test-table [id=8c800c4bc1974b1fb8f9716aa8b9b061]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:45.351487 13283 tablet_service.cc:1467] Processing CreateTablet for tablet 2d907e58e4a74f5680692eb0632ca8e4 (DEFAULT_TABLE table=test-table [id=8c800c4bc1974b1fb8f9716aa8b9b061]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:45.351605 12838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d907e58e4a74f5680692eb0632ca8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:45.351605 13067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d907e58e4a74f5680692eb0632ca8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:45.351750 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d907e58e4a74f5680692eb0632ca8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:45.353919 13438 tablet_bootstrap.cc:492] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13: Bootstrap starting.
I20260810 06:20:45.353919 13437 tablet_bootstrap.cc:492] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5: Bootstrap starting.
I20260810 06:20:45.354007 13440 tablet_bootstrap.cc:492] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d: Bootstrap starting.
I20260810 06:20:45.354719 13438 tablet_bootstrap.cc:654] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:45.354768 13437 tablet_bootstrap.cc:654] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:45.354915 13440 tablet_bootstrap.cc:654] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:45.356352 13438 tablet_bootstrap.cc:492] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13: No bootstrap required, opened a new log
I20260810 06:20:45.356436 13438 ts_tablet_manager.cc:1397] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 06:20:45.356760 13437 tablet_bootstrap.cc:492] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5: No bootstrap required, opened a new log
I20260810 06:20:45.356832 13437 ts_tablet_manager.cc:1397] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 06:20:45.357018 13438 raft_consensus.cc:348] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.357100 13438 raft_consensus.cc:374] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:45.357118 13438 raft_consensus.cc:729] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593684227d124bdda7254e54ed161e13, State: Initialized, Role: FOLLOWER
I20260810 06:20:45.357219 13438 consensus_queue.cc:260] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.357254 13437 raft_consensus.cc:348] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.357360 13437 raft_consensus.cc:374] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:45.357388 13437 raft_consensus.cc:729] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da22030ce30d4474b5a0043fccea9bf5, State: Initialized, Role: FOLLOWER
I20260810 06:20:45.357472 13438 ts_tablet_manager.cc:1428] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 06:20:45.357494 13437 consensus_queue.cc:260] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.357752 13437 ts_tablet_manager.cc:1428] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:45.358548 13440 tablet_bootstrap.cc:492] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d: No bootstrap required, opened a new log
I20260810 06:20:45.358639 13440 ts_tablet_manager.cc:1397] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 06:20:45.359048 13440 raft_consensus.cc:348] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.359146 13440 raft_consensus.cc:374] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:45.359175 13440 raft_consensus.cc:729] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e675b880a284b2a89e319ca900a335d, State: Initialized, Role: FOLLOWER
I20260810 06:20:45.359284 13440 consensus_queue.cc:260] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.360904 13440 ts_tablet_manager.cc:1428] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:45.473763 13517 raft_consensus.cc:482] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:20:45.473896 13517 raft_consensus.cc:504] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.474318 13517 leader_election.cc:288] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da22030ce30d4474b5a0043fccea9bf5 (127.31.204.193:42859), 3e675b880a284b2a89e319ca900a335d (127.31.204.195:42951)
I20260810 06:20:45.474722 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d907e58e4a74f5680692eb0632ca8e4" candidate_uuid: "593684227d124bdda7254e54ed161e13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da22030ce30d4474b5a0043fccea9bf5" is_pre_election: true
I20260810 06:20:45.474712 13321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d907e58e4a74f5680692eb0632ca8e4" candidate_uuid: "593684227d124bdda7254e54ed161e13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e675b880a284b2a89e319ca900a335d" is_pre_election: true
I20260810 06:20:45.474870 13321 raft_consensus.cc:2454] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 593684227d124bdda7254e54ed161e13 in term 0.
I20260810 06:20:45.474879 12873 raft_consensus.cc:2454] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 593684227d124bdda7254e54ed161e13 in term 0.
I20260810 06:20:45.475167 12990 leader_election.cc:302] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593684227d124bdda7254e54ed161e13, da22030ce30d4474b5a0043fccea9bf5; no voters:
I20260810 06:20:45.475338 13517 raft_consensus.cc:2781] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:20:45.475409 13517 raft_consensus.cc:482] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:20:45.475468 13517 raft_consensus.cc:3037] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:45.476608 13517 raft_consensus.cc:504] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.476912 13517 leader_election.cc:288] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [CANDIDATE]: Term 1 election: Requested vote from peers da22030ce30d4474b5a0043fccea9bf5 (127.31.204.193:42859), 3e675b880a284b2a89e319ca900a335d (127.31.204.195:42951)
I20260810 06:20:45.477221 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d907e58e4a74f5680692eb0632ca8e4" candidate_uuid: "593684227d124bdda7254e54ed161e13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da22030ce30d4474b5a0043fccea9bf5"
I20260810 06:20:45.477226 13321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d907e58e4a74f5680692eb0632ca8e4" candidate_uuid: "593684227d124bdda7254e54ed161e13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e675b880a284b2a89e319ca900a335d"
I20260810 06:20:45.477319 12873 raft_consensus.cc:3037] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:45.477322 13321 raft_consensus.cc:3037] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:45.478178 12873 raft_consensus.cc:2454] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593684227d124bdda7254e54ed161e13 in term 1.
I20260810 06:20:45.478305 13321 raft_consensus.cc:2454] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593684227d124bdda7254e54ed161e13 in term 1.
I20260810 06:20:45.478425 12990 leader_election.cc:302] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593684227d124bdda7254e54ed161e13, da22030ce30d4474b5a0043fccea9bf5; no voters:
I20260810 06:20:45.478590 13517 raft_consensus.cc:2781] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:45.478680 13517 raft_consensus.cc:686] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [term 1 LEADER]: Becoming Leader. State: Replica: 593684227d124bdda7254e54ed161e13, State: Running, Role: LEADER
I20260810 06:20:45.478808 13517 consensus_queue.cc:237] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.480223 12617 catalog_manager.cc:5495] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 593684227d124bdda7254e54ed161e13 (127.31.204.194). New cstate: current_term: 1 leader_uuid: "593684227d124bdda7254e54ed161e13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } health_report { overall_health: UNKNOWN } } }
I20260810 06:20:45.511054 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:45.508415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58274 (local address 127.31.204.194:46381)
0810 06:20:45.508539 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:45.508543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:45.508568 (+ 25us) server_negotiation.cc:407] Connection header received
0810 06:20:45.508692 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:45.508694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:45.508736 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:45.508794 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:45.508968 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.509449 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.510224 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.510428 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.510519 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:45.510705 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:45.510768 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:45.510817 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:45.510882 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 06:20:45.510928 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 06:20:45.512692 13321 raft_consensus.cc:1261] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d [term 1 FOLLOWER]: Refusing update from remote peer 593684227d124bdda7254e54ed161e13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:45.512717 12873 raft_consensus.cc:1261] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5 [term 1 FOLLOWER]: Refusing update from remote peer 593684227d124bdda7254e54ed161e13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:45.513118 13535 consensus_queue.cc:1035] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:45.513269 13517 consensus_queue.cc:1035] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:45.519527 12617 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:45.519654 12617 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:45.521636 12617 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c: Sending DeleteTablet for 3 replicas of tablet 2d907e58e4a74f5680692eb0632ca8e4
I20260810 06:20:45.522122 12838 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d907e58e4a74f5680692eb0632ca8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:45 UTC) from {username='oryx', principal='oryx/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:33642
I20260810 06:20:45.522362 13283 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d907e58e4a74f5680692eb0632ca8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:45 UTC) from {username='oryx', principal='oryx/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:40242
I20260810 06:20:45.522436 13556 tablet_replica.cc:330] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5: stopping tablet replica
I20260810 06:20:45.522421 13067 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d907e58e4a74f5680692eb0632ca8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:45 UTC) from {username='oryx', principal='oryx/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:58252
I20260810 06:20:45.522513 13558 tablet_replica.cc:330] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d: stopping tablet replica
I20260810 06:20:45.522555 13556 raft_consensus.cc:2229] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:45.522569 13557 tablet_replica.cc:330] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13: stopping tablet replica
I20260810 06:20:45.522603 13558 raft_consensus.cc:2229] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:45.522675 13556 raft_consensus.cc:2258] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:45.522676 13557 raft_consensus.cc:2229] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:20:45.522701 13558 raft_consensus.cc:2258] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:45.522840 13557 raft_consensus.cc:2258] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:45.522846 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d907e58e4a74f5680692eb0632ca8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:45 UTC) from {username='oryx', principal='oryx/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:40242
I20260810 06:20:45.523320 13558 ts_tablet_manager.cc:1905] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:45.523294 12617 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 06:20:45.523380 13556 ts_tablet_manager.cc:1905] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:45.523383 13557 ts_tablet_manager.cc:1905] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 06:20:45.523643 12617 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:20:45.525622 13556 ts_tablet_manager.cc:1918] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:20:45.525622 13557 ts_tablet_manager.cc:1918] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:20:45.525638 13558 ts_tablet_manager.cc:1918] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:20:45.525688 13557 log.cc:1196] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/2d907e58e4a74f5680692eb0632ca8e4
I20260810 06:20:45.525699 13556 log.cc:1196] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-0/wal/wals/2d907e58e4a74f5680692eb0632ca8e4
I20260810 06:20:45.525704 13558 log.cc:1196] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/2d907e58e4a74f5680692eb0632ca8e4
I20260810 06:20:45.525939 13556 ts_tablet_manager.cc:1939] T 2d907e58e4a74f5680692eb0632ca8e4 P da22030ce30d4474b5a0043fccea9bf5: Deleting consensus metadata
I20260810 06:20:45.525939 13557 ts_tablet_manager.cc:1939] T 2d907e58e4a74f5680692eb0632ca8e4 P 593684227d124bdda7254e54ed161e13: Deleting consensus metadata
I20260810 06:20:45.525980 13558 ts_tablet_manager.cc:1939] T 2d907e58e4a74f5680692eb0632ca8e4 P 3e675b880a284b2a89e319ca900a335d: Deleting consensus metadata
I20260810 06:20:45.526561 12598 catalog_manager.cc:4841] TS 3e675b880a284b2a89e319ca900a335d (127.31.204.195:42951): tablet 2d907e58e4a74f5680692eb0632ca8e4 (table test-table [id=8c800c4bc1974b1fb8f9716aa8b9b061]) successfully deleted
I20260810 06:20:45.526593 12593 catalog_manager.cc:4841] TS 593684227d124bdda7254e54ed161e13 (127.31.204.194:46381): tablet 2d907e58e4a74f5680692eb0632ca8e4 (table test-table [id=8c800c4bc1974b1fb8f9716aa8b9b061]) successfully deleted
W20260810 06:20:45.527052 12598 catalog_manager.cc:4808] TS 3e675b880a284b2a89e319ca900a335d (127.31.204.195:42951): delete failed for tablet 2d907e58e4a74f5680692eb0632ca8e4 because the tablet was not found. No further retry: Not found: Tablet not found: 2d907e58e4a74f5680692eb0632ca8e4
I20260810 06:20:45.527339 12597 catalog_manager.cc:4841] TS da22030ce30d4474b5a0043fccea9bf5 (127.31.204.193:42859): tablet 2d907e58e4a74f5680692eb0632ca8e4 (table test-table [id=8c800c4bc1974b1fb8f9716aa8b9b061]) successfully deleted
I20260810 06:20:45.527868 13067 tablet_service.cc:1467] Processing CreateTablet for tablet 94031803aadb48cdb966119f2597ffad (DEFAULT_TABLE table=test-table [id=a74b507f5bdb45daa53ece0e7d5962b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:45.527913 13282 tablet_service.cc:1467] Processing CreateTablet for tablet 94031803aadb48cdb966119f2597ffad (DEFAULT_TABLE table=test-table [id=a74b507f5bdb45daa53ece0e7d5962b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:45.528103 13067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94031803aadb48cdb966119f2597ffad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:45.528103 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94031803aadb48cdb966119f2597ffad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:45.530241 13438 tablet_bootstrap.cc:492] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13: Bootstrap starting.
I20260810 06:20:45.530241 13440 tablet_bootstrap.cc:492] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d: Bootstrap starting.
I20260810 06:20:45.530227 12838 tablet_service.cc:1467] Processing CreateTablet for tablet 94031803aadb48cdb966119f2597ffad (DEFAULT_TABLE table=test-table [id=a74b507f5bdb45daa53ece0e7d5962b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:20:45.530455 12838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94031803aadb48cdb966119f2597ffad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:45.531067 13438 tablet_bootstrap.cc:654] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:45.531067 13440 tablet_bootstrap.cc:654] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:45.532074 13437 tablet_bootstrap.cc:492] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5: Bootstrap starting.
I20260810 06:20:45.532176 13440 tablet_bootstrap.cc:492] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d: No bootstrap required, opened a new log
I20260810 06:20:45.532251 13438 tablet_bootstrap.cc:492] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13: No bootstrap required, opened a new log
I20260810 06:20:45.532261 13440 ts_tablet_manager.cc:1397] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:45.532325 13438 ts_tablet_manager.cc:1397] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:45.532650 13438 raft_consensus.cc:348] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.532650 13440 raft_consensus.cc:348] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.532759 13440 raft_consensus.cc:374] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:45.532759 13438 raft_consensus.cc:374] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:45.532792 13438 raft_consensus.cc:729] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593684227d124bdda7254e54ed161e13, State: Initialized, Role: FOLLOWER
I20260810 06:20:45.532792 13440 raft_consensus.cc:729] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e675b880a284b2a89e319ca900a335d, State: Initialized, Role: FOLLOWER
I20260810 06:20:45.532830 13437 tablet_bootstrap.cc:654] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:45.532903 13438 consensus_queue.cc:260] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.532903 13440 consensus_queue.cc:260] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.533154 13440 ts_tablet_manager.cc:1428] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:45.533154 13438 ts_tablet_manager.cc:1428] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:45.534164 13437 tablet_bootstrap.cc:492] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5: No bootstrap required, opened a new log
I20260810 06:20:45.534251 13437 ts_tablet_manager.cc:1397] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:45.534627 13437 raft_consensus.cc:348] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.534729 13437 raft_consensus.cc:374] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:45.534766 13437 raft_consensus.cc:729] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da22030ce30d4474b5a0043fccea9bf5, State: Initialized, Role: FOLLOWER
I20260810 06:20:45.534894 13437 consensus_queue.cc:260] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.535143 13437 ts_tablet_manager.cc:1428] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:20:45.746954 13445 raft_consensus.cc:482] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:20:45.747114 13445 raft_consensus.cc:504] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.747550 13445 leader_election.cc:288] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 593684227d124bdda7254e54ed161e13 (127.31.204.194:46381), 3e675b880a284b2a89e319ca900a335d (127.31.204.195:42951)
I20260810 06:20:45.747982 13095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94031803aadb48cdb966119f2597ffad" candidate_uuid: "da22030ce30d4474b5a0043fccea9bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593684227d124bdda7254e54ed161e13" is_pre_election: true
I20260810 06:20:45.747985 13321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94031803aadb48cdb966119f2597ffad" candidate_uuid: "da22030ce30d4474b5a0043fccea9bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e675b880a284b2a89e319ca900a335d" is_pre_election: true
I20260810 06:20:45.748134 13095 raft_consensus.cc:2454] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da22030ce30d4474b5a0043fccea9bf5 in term 0.
I20260810 06:20:45.748121 13321 raft_consensus.cc:2454] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da22030ce30d4474b5a0043fccea9bf5 in term 0.
I20260810 06:20:45.748454 12755 leader_election.cc:302] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593684227d124bdda7254e54ed161e13, da22030ce30d4474b5a0043fccea9bf5; no voters:
I20260810 06:20:45.748622 13445 raft_consensus.cc:2781] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:20:45.748687 13445 raft_consensus.cc:482] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:20:45.748716 13445 raft_consensus.cc:3037] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:45.749725 13445 raft_consensus.cc:504] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.750043 13445 leader_election.cc:288] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [CANDIDATE]: Term 1 election: Requested vote from peers 593684227d124bdda7254e54ed161e13 (127.31.204.194:46381), 3e675b880a284b2a89e319ca900a335d (127.31.204.195:42951)
I20260810 06:20:45.750345 13095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94031803aadb48cdb966119f2597ffad" candidate_uuid: "da22030ce30d4474b5a0043fccea9bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593684227d124bdda7254e54ed161e13"
I20260810 06:20:45.750378 13321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94031803aadb48cdb966119f2597ffad" candidate_uuid: "da22030ce30d4474b5a0043fccea9bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e675b880a284b2a89e319ca900a335d"
I20260810 06:20:45.750450 13095 raft_consensus.cc:3037] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:45.750470 13321 raft_consensus.cc:3037] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:45.751343 13095 raft_consensus.cc:2454] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da22030ce30d4474b5a0043fccea9bf5 in term 1.
I20260810 06:20:45.751348 13321 raft_consensus.cc:2454] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da22030ce30d4474b5a0043fccea9bf5 in term 1.
I20260810 06:20:45.751585 12755 leader_election.cc:302] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593684227d124bdda7254e54ed161e13, da22030ce30d4474b5a0043fccea9bf5; no voters:
I20260810 06:20:45.751740 13445 raft_consensus.cc:2781] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:45.751828 13445 raft_consensus.cc:686] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [term 1 LEADER]: Becoming Leader. State: Replica: da22030ce30d4474b5a0043fccea9bf5, State: Running, Role: LEADER
I20260810 06:20:45.751958 13445 consensus_queue.cc:237] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } }
I20260810 06:20:45.753486 12617 catalog_manager.cc:5495] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to da22030ce30d4474b5a0043fccea9bf5 (127.31.204.193). New cstate: current_term: 1 leader_uuid: "da22030ce30d4474b5a0043fccea9bf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da22030ce30d4474b5a0043fccea9bf5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 42859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 } health_report { overall_health: UNKNOWN } } }
I20260810 06:20:45.772135 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:45.769097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33660 (local address 127.31.204.193:42859)
0810 06:20:45.769349 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:45.769353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:45.769366 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:20:45.769409 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:45.769411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:45.769447 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:45.769517 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:20:45.769748 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.770313 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.771119 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:45.771343 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:45.771437 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:45.771680 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:20:45.771758 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:20:45.771829 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:45.771937 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 06:20:45.771997 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":82,"threads_started":1}
I20260810 06:20:45.777395 13321 raft_consensus.cc:1261] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d [term 1 FOLLOWER]: Refusing update from remote peer da22030ce30d4474b5a0043fccea9bf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:45.777570 13095 raft_consensus.cc:1261] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13 [term 1 FOLLOWER]: Refusing update from remote peer da22030ce30d4474b5a0043fccea9bf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:20:45.777774 13445 consensus_queue.cc:1035] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e675b880a284b2a89e319ca900a335d" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:45.777915 13549 consensus_queue.cc:1035] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593684227d124bdda7254e54ed161e13" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:20:45.805894 12615 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:45.806036 12615 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:20:45.807688 12615 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e75c0fa81ae438f8516c8511f91aa9c: Sending DeleteTablet for 3 replicas of tablet 94031803aadb48cdb966119f2597ffad
I20260810 06:20:45.808161 13067 tablet_service.cc:1514] Processing DeleteTablet for tablet 94031803aadb48cdb966119f2597ffad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:45 UTC) from {username='oryx', principal='oryx/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:58252
I20260810 06:20:45.808379 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet 94031803aadb48cdb966119f2597ffad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:45 UTC) from {username='oryx', principal='oryx/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:40242
I20260810 06:20:45.808550 13557 tablet_replica.cc:330] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13: stopping tablet replica
I20260810 06:20:45.808661 13557 raft_consensus.cc:2229] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:45.808758 13557 raft_consensus.cc:2258] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:45.809031 13558 tablet_replica.cc:330] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d: stopping tablet replica
I20260810 06:20:45.809136 13558 raft_consensus.cc:2229] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:20:45.809249 13558 raft_consensus.cc:2258] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:45.809382 13557 ts_tablet_manager.cc:1905] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:45.809415 12838 tablet_service.cc:1514] Processing DeleteTablet for tablet 94031803aadb48cdb966119f2597ffad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:20:45 UTC) from {username='oryx', principal='oryx/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:33642
I20260810 06:20:45.809566 13556 tablet_replica.cc:330] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5: stopping tablet replica
I20260810 06:20:45.809671 13556 raft_consensus.cc:2229] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:20:45.809823 13556 raft_consensus.cc:2258] T 94031803aadb48cdb966119f2597ffad P da22030ce30d4474b5a0043fccea9bf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:20:45.809834 13558 ts_tablet_manager.cc:1905] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:20:45.810137 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 12736
I20260810 06:20:45.811872 13557 ts_tablet_manager.cc:1918] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:20:45.811939 13557 log.cc:1196] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/94031803aadb48cdb966119f2597ffad
I20260810 06:20:45.812163 13558 ts_tablet_manager.cc:1918] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:20:45.812179 13557 ts_tablet_manager.cc:1939] T 94031803aadb48cdb966119f2597ffad P 593684227d124bdda7254e54ed161e13: Deleting consensus metadata
I20260810 06:20:45.812223 13558 log.cc:1196] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/94031803aadb48cdb966119f2597ffad
I20260810 06:20:45.812448 13558 ts_tablet_manager.cc:1939] T 94031803aadb48cdb966119f2597ffad P 3e675b880a284b2a89e319ca900a335d: Deleting consensus metadata
I20260810 06:20:45.812999 12598 catalog_manager.cc:4841] TS 3e675b880a284b2a89e319ca900a335d (127.31.204.195:42951): tablet 94031803aadb48cdb966119f2597ffad (table test-table [id=a74b507f5bdb45daa53ece0e7d5962b3]) successfully deleted
I20260810 06:20:45.817481 12593 catalog_manager.cc:4841] TS 593684227d124bdda7254e54ed161e13 (127.31.204.194:46381): tablet 94031803aadb48cdb966119f2597ffad (table test-table [id=a74b507f5bdb45daa53ece0e7d5962b3]) successfully deleted
W20260810 06:20:45.825309 12597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.204.193:42859 (error 108)
I20260810 06:20:45.825551 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 12963
I20260810 06:20:45.828902 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:45.828545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:42859 (local address 127.0.0.1:33676)
0810 06:20:45.828756 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:45.828825 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":67,"threads_started":1}
W20260810 06:20:45.829025 12597 catalog_manager.cc:4584] TS da22030ce30d4474b5a0043fccea9bf5 (127.31.204.193:42859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 94031803aadb48cdb966119f2597ffad: Network error: Client connection negotiation failed: client connection to 127.31.204.193:42859: connect: Connection refused (error 111)
I20260810 06:20:45.837105 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 13189
I20260810 06:20:45.845942 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 12567
2026-08-10T06:20:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3723 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:20:45 dist-test-slave-q9h9 krb5kdc[13612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:20:45 dist-test-slave-q9h9 krb5kdc[13612](info): set up 2 sockets
Aug 10 06:20:45 dist-test-slave-q9h9 krb5kdc[13612](info): commencing operation
krb5kdc: starting...
W20260810 06:20:46.992671 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:20:47 dist-test-slave-q9h9 krb5kdc[13612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:20:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:20:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:20:47.104735 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:43935
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:39587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:43935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:47.210093 13637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:47.210315 13637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:47.210353 13637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:47.214214 13637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:47.214260 13637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:47.214277 13637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:47.214306 13637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:47.214334 13637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:20:47.219285 13637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:39587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:43935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:43935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:47.220149 13637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:47.221076 13637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:47.229077 13654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:47.229084 13656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:47.229156 13651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:47.229125 13637 server_base.cc:1034] running on GCE node
I20260810 06:20:47.229830 13637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:47.231019 13637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:47.232188 13637 hybrid_clock.cc:648] HybridClock initialized: now 1786342847232160 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T06:20:47Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1389 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:20:47 dist-test-slave-q9h9 krb5kdc[13665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:20:47 dist-test-slave-q9h9 krb5kdc[13665](info): set up 2 sockets
Aug 10 06:20:47 dist-test-slave-q9h9 krb5kdc[13665](info): commencing operation
krb5kdc: starting...
W20260810 06:20:48.392553 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:20:48 dist-test-slave-q9h9 krb5kdc[13665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:20:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:20:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:20:48.504328 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36927
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:36927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:48.608345 13685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:48.608640 13685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:48.608696 13685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:48.612841 13685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:48.612905 13685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:48.612933 13685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:48.612953 13685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:48.612979 13685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:20:48.618237 13685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:36927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:48.619246 13685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:48.620255 13685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:48.628536 13702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:48.628537 13691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:48.628612 13695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:48.628723 13685 server_base.cc:1034] running on GCE node
I20260810 06:20:48.629278 13685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:48.630383 13685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:48.631553 13685 hybrid_clock.cc:648] HybridClock initialized: now 1786342848631537 us; error 26 us; skew 500 ppm
Aug 10 06:20:48 dist-test-slave-q9h9 krb5kdc[13665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342848, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:48.634163 13685 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:20:48.635001 13685 webserver.cc:460] Webserver started at http://127.31.204.254:39059/ using document root <none> and password file <none>
I20260810 06:20:48.635493 13685 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:48.635550 13685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:48.635710 13685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:48.637609 13685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "701f5fd1369d4d908b4e2719ab28012d"
format_stamp: "Formatted at 2026-08-10 06:20:48 on dist-test-slave-q9h9"
server_key: "03eecc02b6613bb0e712f74aab738fe4"
server_key_iv: "64ff2a4c2df052e1f8d281aa4936e30a"
server_key_version: "encryptionkey@0"
I20260810 06:20:48.638135 13685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "701f5fd1369d4d908b4e2719ab28012d"
format_stamp: "Formatted at 2026-08-10 06:20:48 on dist-test-slave-q9h9"
server_key: "03eecc02b6613bb0e712f74aab738fe4"
server_key_iv: "64ff2a4c2df052e1f8d281aa4936e30a"
server_key_version: "encryptionkey@0"
I20260810 06:20:48.641487 13685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:20:48.643697 13714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:48.644785 13685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:48.644879 13685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "701f5fd1369d4d908b4e2719ab28012d"
format_stamp: "Formatted at 2026-08-10 06:20:48 on dist-test-slave-q9h9"
server_key: "03eecc02b6613bb0e712f74aab738fe4"
server_key_iv: "64ff2a4c2df052e1f8d281aa4936e30a"
server_key_version: "encryptionkey@0"
I20260810 06:20:48.644955 13685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:48.662087 13685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:48.664569 13685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:48.664707 13685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:48.671118 13685 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:36927
I20260810 06:20:48.671132 13800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:36927 every 8 connection(s)
I20260810 06:20:48.671954 13685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:20:48.674778 13801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:20:48.679139 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 13685
I20260810 06:20:48.679289 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:20:48.679592 32563 external_mini_cluster.cc:1428] Setting key 29c4e6289c4b119acd38dd608159a5ce
I20260810 06:20:48.680133 13801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d: Bootstrap starting.
I20260810 06:20:48.682417 13801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d: Neither blocks nor log segments found. Creating new log.
I20260810 06:20:48.683110 13801 log.cc:824] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:48.685020 13801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d: No bootstrap required, opened a new log
Aug 10 06:20:48 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:48.688823 13801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701f5fd1369d4d908b4e2719ab28012d" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36927 } }
I20260810 06:20:48.689060 13801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:20:48.689123 13801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 701f5fd1369d4d908b4e2719ab28012d, State: Initialized, Role: FOLLOWER
I20260810 06:20:48.689564 13801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701f5fd1369d4d908b4e2719ab28012d" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36927 } }
I20260810 06:20:48.689757 13801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:20:48.689836 13801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:20:48.689931 13801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:20:48.690902 13801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701f5fd1369d4d908b4e2719ab28012d" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36927 } }
I20260810 06:20:48.691231 13801 leader_election.cc:302] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 701f5fd1369d4d908b4e2719ab28012d; no voters:
I20260810 06:20:48.691522 13801 leader_election.cc:288] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:20:48.691589 13810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:20:48.691792 13810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [term 1 LEADER]: Becoming Leader. State: Replica: 701f5fd1369d4d908b4e2719ab28012d, State: Running, Role: LEADER
I20260810 06:20:48.692116 13810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701f5fd1369d4d908b4e2719ab28012d" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36927 } }
I20260810 06:20:48.692704 13801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:48.693570 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:48.680874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50034 (local address 127.31.204.254:36927)
0810 06:20:48.681278 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:48.681292 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:48.681325 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:20:48.681907 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:48.681922 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:48.682277 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:48.682557 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:48.683344 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:48.684614 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:48.685483 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:48.686163 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:48.686327 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:48.688657 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:48.688663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:48.688677 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:48.688706 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:48.690609 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:48.691006 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:48.691012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:48.691017 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:48.691150 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:48.691410 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:48.691413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:48.691415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:48.691870 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:48.692080 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:48.692439 (+ 359us) server_negotiation.cc:299] Negotiation successful
0810 06:20:48.692730 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":126,"threads_started":1}
I20260810 06:20:48.693782 13810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 701f5fd1369d4d908b4e2719ab28012d. Latest consensus state: current_term: 1 leader_uuid: "701f5fd1369d4d908b4e2719ab28012d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701f5fd1369d4d908b4e2719ab28012d" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36927 } } }
I20260810 06:20:48.694919 13810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:48.693635 13811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "701f5fd1369d4d908b4e2719ab28012d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701f5fd1369d4d908b4e2719ab28012d" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36927 } } }
I20260810 06:20:48.695128 13811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [sys.catalog]: This master's current role is: LEADER
W20260810 06:20:48.697124 13831 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:20:48.697196 13831 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:20:48.697271 13833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:48.697955 13833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:48.702486 13833 catalog_manager.cc:1347] Generated new cluster ID: cf78ac9e953a409181e08e4b52bda2ef
I20260810 06:20:48.702543 13833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:48.712719 13833 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:20:48.713828 13833 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:48.718571 13833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d: Generated new TSK 0
I20260810 06:20:48.719275 13833 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
I20260810 06:20:48.752247 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36927
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:48.859449 13844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:48.859659 13844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:48.859699 13844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:48.863626 13844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:48.863672 13844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:48.863737 13844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:20:48.868664 13844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:48.869750 13844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:48.870747 13844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:48.879671 13852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:48.879693 13854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:48.879716 13862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:48.879891 13844 server_base.cc:1034] running on GCE node
I20260810 06:20:48.880280 13844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:48.880858 13844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:48.882010 13844 hybrid_clock.cc:648] HybridClock initialized: now 1786342848881993 us; error 28 us; skew 500 ppm
Aug 10 06:20:48 dist-test-slave-q9h9 krb5kdc[13665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342848, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:48.884518 13844 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:20:48.885295 13844 webserver.cc:460] Webserver started at http://127.31.204.193:44683/ using document root <none> and password file <none>
I20260810 06:20:48.885777 13844 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:48.885838 13844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:48.886050 13844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:48.887972 13844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "dcacaae0f958493781fe78b046b9386d"
format_stamp: "Formatted at 2026-08-10 06:20:48 on dist-test-slave-q9h9"
server_key: "d6dab41f8cc17a478776cb1c05e8db01"
server_key_iv: "69c85f97583cb80677ee18bfb7099297"
server_key_version: "encryptionkey@0"
I20260810 06:20:48.888561 13844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "dcacaae0f958493781fe78b046b9386d"
format_stamp: "Formatted at 2026-08-10 06:20:48 on dist-test-slave-q9h9"
server_key: "d6dab41f8cc17a478776cb1c05e8db01"
server_key_iv: "69c85f97583cb80677ee18bfb7099297"
server_key_version: "encryptionkey@0"
I20260810 06:20:48.892196 13844 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:20:48.894596 13874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:48.895762 13844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:48.895879 13844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "dcacaae0f958493781fe78b046b9386d"
format_stamp: "Formatted at 2026-08-10 06:20:48 on dist-test-slave-q9h9"
server_key: "d6dab41f8cc17a478776cb1c05e8db01"
server_key_iv: "69c85f97583cb80677ee18bfb7099297"
server_key_version: "encryptionkey@0"
I20260810 06:20:48.895968 13844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:48.908064 13844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:48.910583 13844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:48.910749 13844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:48.911279 13844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:48.912030 13844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:48.912076 13844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:48.912127 13844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:48.912155 13844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:48.920476 13844 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:43833
I20260810 06:20:48.920498 14048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:43833 every 8 connection(s)
I20260810 06:20:48.921310 13844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:20:48.927548 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 13844
I20260810 06:20:48.927672 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:20:48.927946 32563 external_mini_cluster.cc:1428] Setting key fcf09e35a6eb506dad5ce1362fc2f12b
Aug 10 06:20:48 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342848, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:48.934235 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:48.922952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:57017 (local address 127.31.204.254:36927)
0810 06:20:48.923074 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:48.923077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:48.924018 (+ 941us) server_negotiation.cc:407] Connection header received
0810 06:20:48.924876 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:48.924880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:48.924940 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:48.925011 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:48.926110 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:48.926649 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:48.927768 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:48.928092 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:48.928202 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:48.930869 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:48.930873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:48.930875 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:48.930902 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:48.932252 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:48.932772 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:48.932775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:48.932777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:48.932865 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:48.933136 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:48.933139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:48.933141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:48.933362 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:48.933452 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:48.933902 (+ 450us) server_negotiation.cc:299] Negotiation successful
0810 06:20:48.934064 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 06:20:48.934921 14052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:48.923260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.193:57017)
0810 06:20:48.923879 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:48.923917 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:48.924674 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:48.925183 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:48.925190 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:48.925518 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:48.925966 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:48.925979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:48.926771 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:48.926773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:48.927639 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:48.927650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:48.928268 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:48.928273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:48.928469 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:48.929043 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:48.929062 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:48.930708 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:48.932362 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:48.932369 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:48.932382 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:48.932681 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:48.932942 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:48.932944 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:48.932945 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:48.933062 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:48.933452 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:48.933456 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:48.933673 (+ 217us) client_negotiation.cc:769] Sending connection context
0810 06:20:48.933818 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 06:20:48.934092 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":114,"threads_started":1}
I20260810 06:20:48.935985 14049 heartbeater.cc:347] Connected to a master server at 127.31.204.254:36927
I20260810 06:20:48.936245 14049 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:48.936812 14049 heartbeater.cc:511] Master 127.31.204.254:36927 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.204.194@KRBTEST.COM; defaulting to no policy
I20260810 06:20:48.938385 13743 ts_manager.cc:194] Registered new tserver with Master: dcacaae0f958493781fe78b046b9386d (127.31.204.193:43833)
I20260810 06:20:48.939582 13743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:57017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
I20260810 06:20:48.954384 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36927
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:49.059556 14056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:49.059770 14056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:49.059808 14056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:49.063962 14056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:49.064018 14056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:49.064083 14056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:20:49.068979 14056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:49.069842 14056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:49.070883 14056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:49.079778 14068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:49.079841 14067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:49.079787 14075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:49.080024 14056 server_base.cc:1034] running on GCE node
I20260810 06:20:49.080391 14056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:49.080910 14056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:49.082070 14056 hybrid_clock.cc:648] HybridClock initialized: now 1786342849082044 us; error 36 us; skew 500 ppm
Aug 10 06:20:49 dist-test-slave-q9h9 krb5kdc[13665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342849, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:49.084474 14056 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:20:49.085213 14056 webserver.cc:460] Webserver started at http://127.31.204.194:41301/ using document root <none> and password file <none>
I20260810 06:20:49.085707 14056 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:49.085777 14056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:49.085968 14056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:49.087798 14056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "f00451c18ed64644a2e4d8486e1a35f6"
format_stamp: "Formatted at 2026-08-10 06:20:49 on dist-test-slave-q9h9"
server_key: "9a3b8ffdd81248b762bfc9c5305714e5"
server_key_iv: "3db7fa17e6cf11414c05e21edf12d971"
server_key_version: "encryptionkey@0"
I20260810 06:20:49.088414 14056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "f00451c18ed64644a2e4d8486e1a35f6"
format_stamp: "Formatted at 2026-08-10 06:20:49 on dist-test-slave-q9h9"
server_key: "9a3b8ffdd81248b762bfc9c5305714e5"
server_key_iv: "3db7fa17e6cf11414c05e21edf12d971"
server_key_version: "encryptionkey@0"
I20260810 06:20:49.091861 14056 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:20:49.094259 14086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:49.095338 14056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:49.095439 14056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "f00451c18ed64644a2e4d8486e1a35f6"
format_stamp: "Formatted at 2026-08-10 06:20:49 on dist-test-slave-q9h9"
server_key: "9a3b8ffdd81248b762bfc9c5305714e5"
server_key_iv: "3db7fa17e6cf11414c05e21edf12d971"
server_key_version: "encryptionkey@0"
I20260810 06:20:49.095527 14056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:49.107683 14056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:49.110047 14056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:49.110205 14056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:49.110697 14056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:49.111419 14056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:49.111464 14056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:49.111513 14056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:49.111539 14056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:49.119452 14056 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:37633
I20260810 06:20:49.119477 14271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:37633 every 8 connection(s)
I20260810 06:20:49.120184 14056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
Aug 10 06:20:49 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342849, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:49.129369 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 14056
I20260810 06:20:49.129479 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:20:49.129784 32563 external_mini_cluster.cc:1428] Setting key b011a5d7f238629d4895e3ef1a7d3ecf
I20260810 06:20:49.132153 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:49.121690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:59823 (local address 127.31.204.254:36927)
0810 06:20:49.121856 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:49.121859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:49.122818 (+ 959us) server_negotiation.cc:407] Connection header received
0810 06:20:49.123606 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:49.123609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:49.123664 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:49.123736 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:49.124780 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.125285 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.126399 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.126614 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.126702 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:49.128790 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:49.128793 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:49.128795 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:49.128817 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:49.130234 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:49.130740 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:49.130743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:49.130745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:49.130830 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:49.131167 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:49.131170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:49.131172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:49.131355 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:49.131440 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:49.131883 (+ 443us) server_negotiation.cc:299] Negotiation successful
0810 06:20:49.132005 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 06:20:49.132812 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:49.122054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.194:59823)
0810 06:20:49.122666 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:49.122703 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:49.123417 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:49.123905 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:49.123911 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:49.124214 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:49.124633 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:49.124644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.125398 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.125401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:49.126285 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:49.126295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.126747 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.126751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:49.126898 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:49.127362 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:49.127378 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:49.128630 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:49.130343 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:49.130349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:49.130362 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:49.130647 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:49.130911 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:49.130913 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:49.130915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:49.131079 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:49.131454 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:49.131460 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:49.131661 (+ 201us) client_negotiation.cc:769] Sending connection context
0810 06:20:49.131821 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 06:20:49.132092 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":106,"threads_started":1}
I20260810 06:20:49.133761 14276 heartbeater.cc:347] Connected to a master server at 127.31.204.254:36927
I20260810 06:20:49.133961 14276 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:49.134450 14276 heartbeater.cc:511] Master 127.31.204.254:36927 requested a full tablet report, sending...
I20260810 06:20:49.135375 13743 ts_manager.cc:194] Registered new tserver with Master: f00451c18ed64644a2e4d8486e1a35f6 (127.31.204.194:37633)
I20260810 06:20:49.136111 13743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:59823
WARNING: no policy specified for kudu/127.31.204.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
I20260810 06:20:49.154395 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36927
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:49.259754 14283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:49.259961 14283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:49.260004 14283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:49.263906 14283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:49.263955 14283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:49.264015 14283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:20:49.268852 14283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:49.269663 14283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:49.270563 14283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:49.279011 14288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:49.279031 14290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:49.279042 14293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:49.279227 14283 server_base.cc:1034] running on GCE node
I20260810 06:20:49.279688 14283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:49.280223 14283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:49.281381 14283 hybrid_clock.cc:648] HybridClock initialized: now 1786342849281366 us; error 32 us; skew 500 ppm
Aug 10 06:20:49 dist-test-slave-q9h9 krb5kdc[13665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342849, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:49.283790 14283 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:20:49.284550 14283 webserver.cc:460] Webserver started at http://127.31.204.195:46077/ using document root <none> and password file <none>
I20260810 06:20:49.285017 14283 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:49.285079 14283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:49.285246 14283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:20:49.287122 14283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "53cae77a14084637b49c1080fdd82ce7"
format_stamp: "Formatted at 2026-08-10 06:20:49 on dist-test-slave-q9h9"
server_key: "ede8f224f9fe4bf43c16509e68674645"
server_key_iv: "9cc7681dab4d966a66110cf21c6d16b7"
server_key_version: "encryptionkey@0"
I20260810 06:20:49.287627 14283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "53cae77a14084637b49c1080fdd82ce7"
format_stamp: "Formatted at 2026-08-10 06:20:49 on dist-test-slave-q9h9"
server_key: "ede8f224f9fe4bf43c16509e68674645"
server_key_iv: "9cc7681dab4d966a66110cf21c6d16b7"
server_key_version: "encryptionkey@0"
I20260810 06:20:49.290917 14283 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:20:49.293001 14305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:49.294155 14283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:49.294257 14283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "53cae77a14084637b49c1080fdd82ce7"
format_stamp: "Formatted at 2026-08-10 06:20:49 on dist-test-slave-q9h9"
server_key: "ede8f224f9fe4bf43c16509e68674645"
server_key_iv: "9cc7681dab4d966a66110cf21c6d16b7"
server_key_version: "encryptionkey@0"
I20260810 06:20:49.294366 14283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:49.310506 14283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:49.312865 14283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:49.313017 14283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:49.313514 14283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:49.314266 14283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:49.314316 14283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:49.314354 14283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:49.314383 14283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:49.322058 14283 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:39665
I20260810 06:20:49.322108 14484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:39665 every 8 connection(s)
I20260810 06:20:49.322824 14283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:20:49.329619 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 14283
I20260810 06:20:49.329738 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:20:49.329986 32563 external_mini_cluster.cc:1428] Setting key c7c2d80ed3d461de163c7ab4424d6c6f
Aug 10 06:20:49 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342849, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:20:49.334076 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:49.324345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:36127 (local address 127.31.204.254:36927)
0810 06:20:49.324498 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:49.324503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:49.325396 (+ 893us) server_negotiation.cc:407] Connection header received
0810 06:20:49.326168 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:49.326170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:49.326214 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:49.326282 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:49.327236 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.327736 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.328737 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.328940 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.329002 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:49.331357 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:49.331360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:49.331361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:49.331385 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:49.332555 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:49.332979 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:49.332981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:49.332983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:49.333046 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:49.333283 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:49.333284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:49.333285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:49.333428 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:49.333495 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:49.333852 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 06:20:49.333938 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 06:20:49.334748 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:49.324681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.195:36127)
0810 06:20:49.325256 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:49.325304 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:49.325976 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:49.326419 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:49.326425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:49.326726 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:49.327129 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:49.327141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.327830 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.327832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:49.328653 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:49.328661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.329035 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.329037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:49.329162 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:49.329650 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:49.329664 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:49.331173 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:20:49.332653 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:49.332658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:49.332667 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:49.332906 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:49.333104 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:20:49.333106 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:20:49.333107 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:20:49.333219 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:20:49.333494 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:20:49.333499 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:20:49.333686 (+ 187us) client_negotiation.cc:769] Sending connection context
0810 06:20:49.333823 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 06:20:49.334062 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":105,"threads_started":1}
I20260810 06:20:49.335670 14486 heartbeater.cc:347] Connected to a master server at 127.31.204.254:36927
I20260810 06:20:49.335848 14486 heartbeater.cc:464] Registering TS with master...
I20260810 06:20:49.336210 14486 heartbeater.cc:511] Master 127.31.204.254:36927 requested a full tablet report, sending...
I20260810 06:20:49.337025 13743 ts_manager.cc:194] Registered new tserver with Master: 53cae77a14084637b49c1080fdd82ce7 (127.31.204.195:39665)
I20260810 06:20:49.337512 13743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:36127
I20260810 06:20:49.343222 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.193.keytab.
Entry for principal oryx/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.193.keytab.
WARNING: no policy specified for oryx/127.31.204.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.204.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.194.keytab.
Entry for principal oryx/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.194.keytab.
WARNING: no policy specified for oryx/127.31.204.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.204.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.195.keytab.
Entry for principal oryx/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.195.keytab.
I20260810 06:20:49.401580 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 13844
I20260810 06:20:49.416138 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 14056
I20260810 06:20:49.425112 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 14283
I20260810 06:20:49.434204 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 13685
I20260810 06:20:49.441994 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36927
--webserver_interface=127.31.204.254
--webserver_port=39059
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:36927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:49.544816 14504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:49.545023 14504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:49.545063 14504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:49.548888 14504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:20:49.548933 14504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:49.548951 14504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:49.548980 14504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:20:49.549007 14504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:20:49.553901 14504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:36927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=39059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:49.554752 14504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:49.555661 14504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:49.563247 14517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:49.563169 14514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:49.563431 14504 server_base.cc:1034] running on GCE node
W20260810 06:20:49.563184 14515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:49.563982 14504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:49.564919 14504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:49.566063 14504 hybrid_clock.cc:648] HybridClock initialized: now 1786342849566042 us; error 36 us; skew 500 ppm
Aug 10 06:20:49 dist-test-slave-q9h9 krb5kdc[13665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342849, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:49.568512 14504 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:20:49.569252 14504 webserver.cc:460] Webserver started at http://127.31.204.254:39059/ using document root <none> and password file <none>
I20260810 06:20:49.569710 14504 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:49.569796 14504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:49.573109 14504 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:49.574981 14529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:49.576031 14504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:20:49.576134 14504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "701f5fd1369d4d908b4e2719ab28012d"
format_stamp: "Formatted at 2026-08-10 06:20:48 on dist-test-slave-q9h9"
server_key: "03eecc02b6613bb0e712f74aab738fe4"
server_key_iv: "64ff2a4c2df052e1f8d281aa4936e30a"
server_key_version: "encryptionkey@0"
I20260810 06:20:49.576608 14504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:49.591601 14504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:49.594259 14504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:49.594393 14504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:49.600605 14504 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:36927
I20260810 06:20:49.600701 14612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:36927 every 8 connection(s)
I20260810 06:20:49.601513 14504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:20:49.605553 14613 sys_catalog.cc:263] Verifying existing consensus state
I20260810 06:20:49.607185 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 14504
I20260810 06:20:49.607380 14613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d: Bootstrap starting.
I20260810 06:20:49.607688 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:43833
--local_ip_for_outbound_sockets=127.31.204.193
--tserver_master_addrs=127.31.204.254:36927
--webserver_port=44683
--webserver_interface=127.31.204.193
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.204.193
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:20:49.630693 14613 log.cc:824] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d: Log is configured to *not* fsync() on all Append() calls
I20260810 06:20:49.637530 14613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:20:49.637862 14613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d: Bootstrap complete.
I20260810 06:20:49.642248 14613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701f5fd1369d4d908b4e2719ab28012d" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36927 } }
I20260810 06:20:49.642634 14613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 701f5fd1369d4d908b4e2719ab28012d, State: Initialized, Role: FOLLOWER
I20260810 06:20:49.643088 14613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701f5fd1369d4d908b4e2719ab28012d" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36927 } }
I20260810 06:20:49.643211 14613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:20:49.643282 14613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:20:49.643380 14613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [term 1 FOLLOWER]: Advancing to term 2
I20260810 06:20:49.645192 14613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701f5fd1369d4d908b4e2719ab28012d" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36927 } }
I20260810 06:20:49.645489 14613 leader_election.cc:302] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 701f5fd1369d4d908b4e2719ab28012d; no voters:
I20260810 06:20:49.645799 14613 leader_election.cc:288] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 06:20:49.645920 14619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [term 2 FOLLOWER]: Leader election won for term 2
I20260810 06:20:49.646139 14619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [term 2 LEADER]: Becoming Leader. State: Replica: 701f5fd1369d4d908b4e2719ab28012d, State: Running, Role: LEADER
I20260810 06:20:49.646426 14619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701f5fd1369d4d908b4e2719ab28012d" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36927 } }
I20260810 06:20:49.646999 14613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:20:49.647357 14621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 701f5fd1369d4d908b4e2719ab28012d. Latest consensus state: current_term: 2 leader_uuid: "701f5fd1369d4d908b4e2719ab28012d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701f5fd1369d4d908b4e2719ab28012d" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36927 } } }
I20260810 06:20:49.647475 14621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:49.647704 14620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "701f5fd1369d4d908b4e2719ab28012d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701f5fd1369d4d908b4e2719ab28012d" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36927 } } }
I20260810 06:20:49.647811 14620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d [sys.catalog]: This master's current role is: LEADER
I20260810 06:20:49.647878 14624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:20:49.650962 14624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:20:49.651700 14624 catalog_manager.cc:1259] Loaded cluster ID: cf78ac9e953a409181e08e4b52bda2ef
I20260810 06:20:49.651757 14624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:20:49.654084 14624 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:20:49.654722 14624 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 701f5fd1369d4d908b4e2719ab28012d: Loaded TSK: 0
I20260810 06:20:49.655907 14624 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 06:20:49.727039 14615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:49.727269 14615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:49.727339 14615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:49.731225 14615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:49.731283 14615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:49.731365 14615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:20:49.736375 14615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.193.keytab
--principal=oryx/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:43833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=44683
--tserver_master_addrs=127.31.204.254:36927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:49.737244 14615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:49.738194 14615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:49.748391 14657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:49.748452 14652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:49.748653 14615 server_base.cc:1034] running on GCE node
W20260810 06:20:49.748816 14653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:49.749289 14615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:49.749868 14615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:49.751041 14615 hybrid_clock.cc:648] HybridClock initialized: now 1786342849751007 us; error 51 us; skew 500 ppm
Aug 10 06:20:49 dist-test-slave-q9h9 krb5kdc[13665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342849, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:49.753708 14615 init.cc:377] Logged in from keytab as oryx/127.31.204.193@KRBTEST.COM (short username oryx)
I20260810 06:20:49.754611 14615 webserver.cc:460] Webserver started at http://127.31.204.193:44683/ using document root <none> and password file <none>
I20260810 06:20:49.755137 14615 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:49.755218 14615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:49.758829 14615 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:49.760931 14666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:49.762131 14615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:20:49.762254 14615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "dcacaae0f958493781fe78b046b9386d"
format_stamp: "Formatted at 2026-08-10 06:20:48 on dist-test-slave-q9h9"
server_key: "d6dab41f8cc17a478776cb1c05e8db01"
server_key_iv: "69c85f97583cb80677ee18bfb7099297"
server_key_version: "encryptionkey@0"
I20260810 06:20:49.762756 14615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:49.785002 14615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:49.787616 14615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:49.787775 14615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:49.788300 14615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:49.789091 14615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:49.789142 14615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:49.789193 14615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:49.789217 14615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:49.798251 14615 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:43833
I20260810 06:20:49.798282 14839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:43833 every 8 connection(s)
I20260810 06:20:49.799202 14615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:20:49.805526 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 14615
I20260810 06:20:49.805924 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:37633
--local_ip_for_outbound_sockets=127.31.204.194
--tserver_master_addrs=127.31.204.254:36927
--webserver_port=41301
--webserver_interface=127.31.204.194
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.204.194
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 06:20:49 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:49 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 06:20:49.811379 14843 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:49.801082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:54619 (local address 127.31.204.254:36927)
0810 06:20:49.801528 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:49.801550 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:49.801992 (+ 442us) server_negotiation.cc:407] Connection header received
0810 06:20:49.802942 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:49.802953 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:49.803232 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:49.803497 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:49.804444 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.805413 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.806677 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.807164 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.807334 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:49.810750 (+ 3416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.193:54619: BlockingRecv error: recv got EOF from 127.31.204.193:54619 (error 108)
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":99,"threads_started":1}
I20260810 06:20:49.811451 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:49.801206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.193:54619)
0810 06:20:49.801842 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:49.801879 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:49.802616 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:49.803563 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:49.803568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:49.803883 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:49.804308 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:49.804321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.805570 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.805573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:49.806529 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:49.806541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.807330 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.807335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:49.807518 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:49.808036 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:49.808052 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:49.810555 (+ 2503us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":145,"threads_started":1}
W20260810 06:20:49.811962 14841 heartbeater.cc:644] Failed to heartbeat to 127.31.204.254:36927 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.204.254:36927: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Aug 10 06:20:49 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:49 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:49.817047 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:49.812369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.193:51193)
0810 06:20:49.812514 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:49.812528 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:49.812617 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:49.812842 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:49.812845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:49.812992 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:49.813088 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:49.813094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.813800 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.813803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:49.814366 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:49.814375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.814819 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.814823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:49.814930 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:49.815354 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:49.815366 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:49.816928 (+ 1562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90}
W20260810 06:20:49.817184 14843 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:49.812471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:51193 (local address 127.31.204.254:36927)
0810 06:20:49.812593 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:49.812598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:49.812613 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:20:49.812701 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:49.812704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:49.812748 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:49.812818 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:49.813200 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.813691 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.814477 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.814692 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.814782 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:49.817080 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.193:51193: BlockingRecv error: recv got EOF from 127.31.204.193:51193 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 06:20:49 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:49 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:49.822387 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:49.817729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.193:35971)
0810 06:20:49.817835 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:49.817849 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:49.817912 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:49.818142 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:49.818144 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:49.818262 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:49.818328 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:49.818333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.819251 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.819253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:49.819738 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:49.819745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.820247 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.820249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:49.820303 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:49.820706 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:49.820718 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:49.822284 (+ 1566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 06:20:49.822494 14843 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:49.817805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:35971 (local address 127.31.204.254:36927)
0810 06:20:49.817905 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:49.817909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:49.817922 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:20:49.817996 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:49.817998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:49.818056 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:49.818125 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:49.818420 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.819167 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.819828 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.820153 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.820235 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:49.822397 (+ 2162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.193:35971: BlockingRecv error: recv got EOF from 127.31.204.193:35971 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 06:20:49.822669 14841 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 06:20:49.914590 14846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:49.914798 14846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:49.914868 14846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:49.918749 14846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:49.918797 14846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:49.918870 14846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:20:49.923697 14846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.194.keytab
--principal=oryx/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:37633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=41301
--tserver_master_addrs=127.31.204.254:36927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:49.924538 14846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:49.925429 14846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:49.933795 14854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:49.933797 14852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:49.933827 14851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:49.934110 14846 server_base.cc:1034] running on GCE node
I20260810 06:20:49.934463 14846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:49.935038 14846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:49.936199 14846 hybrid_clock.cc:648] HybridClock initialized: now 1786342849936177 us; error 29 us; skew 500 ppm
Aug 10 06:20:49 dist-test-slave-q9h9 krb5kdc[13665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342849, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:49.938663 14846 init.cc:377] Logged in from keytab as oryx/127.31.204.194@KRBTEST.COM (short username oryx)
I20260810 06:20:49.939380 14846 webserver.cc:460] Webserver started at http://127.31.204.194:41301/ using document root <none> and password file <none>
I20260810 06:20:49.939836 14846 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:49.939904 14846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:49.943321 14846 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:49.945245 14868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:49.946345 14846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:49.946437 14846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "f00451c18ed64644a2e4d8486e1a35f6"
format_stamp: "Formatted at 2026-08-10 06:20:49 on dist-test-slave-q9h9"
server_key: "9a3b8ffdd81248b762bfc9c5305714e5"
server_key_iv: "3db7fa17e6cf11414c05e21edf12d971"
server_key_version: "encryptionkey@0"
I20260810 06:20:49.946902 14846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:49.961421 14846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:49.963899 14846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:49.964030 14846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:49.964481 14846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:49.965171 14846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:49.965215 14846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:49.965250 14846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:49.965271 14846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:49.973331 14846 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:37633
I20260810 06:20:49.973388 15046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:37633 every 8 connection(s)
I20260810 06:20:49.974244 14846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
Aug 10 06:20:49 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:49.982697 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 14846
Aug 10 06:20:49 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:49.983131 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:39665
--local_ip_for_outbound_sockets=127.31.204.195
--tserver_master_addrs=127.31.204.254:36927
--webserver_port=46077
--webserver_interface=127.31.204.195
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.204.195
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:20:49.984016 14843 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:49.975875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:55919 (local address 127.31.204.254:36927)
0810 06:20:49.976006 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:49.976011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:49.976774 (+ 763us) server_negotiation.cc:407] Connection header received
0810 06:20:49.977887 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:49.977889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:49.977939 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:49.978007 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:49.979012 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.979541 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.980651 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.980890 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.980962 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:49.983897 (+ 2935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.194:55919: BlockingRecv error: recv got EOF from 127.31.204.194:55919 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 06:20:49.984522 15051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:49.976232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.194:55919)
0810 06:20:49.976633 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:49.976672 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:49.977709 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:49.978158 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:49.978164 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:49.978474 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:49.978903 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:49.978915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.979639 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.979641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:49.980547 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:49.980557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.980998 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.981000 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:49.981124 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:49.981572 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:49.981586 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:49.983770 (+ 2184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":103,"threads_started":1}
W20260810 06:20:49.984963 15047 heartbeater.cc:644] Failed to heartbeat to 127.31.204.254:36927 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.204.254:36927: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Aug 10 06:20:49 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:49 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:49.990008 15051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:49.985321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.194:35427)
0810 06:20:49.985532 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:49.985545 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:49.985622 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:49.985823 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:49.985825 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:49.985961 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:49.986071 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:49.986078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.986745 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.986749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:49.987262 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:49.987271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.987819 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.987821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:49.987885 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:49.988285 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:49.988295 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:49.989895 (+ 1600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155}
W20260810 06:20:49.990137 14843 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:49.985426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:35427 (local address 127.31.204.254:36927)
0810 06:20:49.985532 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:49.985536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:49.985585 (+ 49us) server_negotiation.cc:407] Connection header received
0810 06:20:49.985696 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:49.985699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:49.985737 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:49.985798 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:49.986178 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.986645 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.987390 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.987715 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.987818 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:49.990000 (+ 2182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.194:35427: BlockingRecv error: recv got EOF from 127.31.204.194:35427 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 10 06:20:49 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:49 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:49.994997 15051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:49.990584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.194:35663)
0810 06:20:49.990795 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:49.990808 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:49.990884 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:49.991072 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:49.991075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:49.991203 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:49.991277 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:49.991281 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.991919 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.991922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:49.992405 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:49.992413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.992791 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.992794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:49.992864 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:49.993262 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:49.993272 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:49.994893 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156}
W20260810 06:20:49.995102 14843 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:49.990691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:35663 (local address 127.31.204.254:36927)
0810 06:20:49.990797 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:49.990800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:49.990848 (+ 48us) server_negotiation.cc:407] Connection header received
0810 06:20:49.990955 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:49.990958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:49.990990 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:49.991048 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:49.991372 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.991828 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.992502 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:49.992695 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:49.992781 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:49.995010 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.194:35663: BlockingRecv error: recv got EOF from 127.31.204.194:35663 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 06:20:49.995253 15047 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 06:20:50.089510 15054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:20:50.089731 15054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:20:50.089788 15054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:20:50.093644 15054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:20:50.093695 15054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:20:50.093771 15054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:20:50.098875 15054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/krb5kdc/oryx_127.31.204.195.keytab
--principal=oryx/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:39665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=46077
--tserver_master_addrs=127.31.204.254:36927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:20:50.099764 15054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:20:50.100665 15054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:20:50.109285 15064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:50.109423 15054 server_base.cc:1034] running on GCE node
W20260810 06:20:50.109265 15062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:20:50.109263 15061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:20:50.109874 15054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:20:50.110471 15054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:20:50.111629 15054 hybrid_clock.cc:648] HybridClock initialized: now 1786342850111600 us; error 39 us; skew 500 ppm
Aug 10 06:20:50 dist-test-slave-q9h9 krb5kdc[13665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342850, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:20:50.114154 15054 init.cc:377] Logged in from keytab as oryx/127.31.204.195@KRBTEST.COM (short username oryx)
I20260810 06:20:50.114913 15054 webserver.cc:460] Webserver started at http://127.31.204.195:46077/ using document root <none> and password file <none>
I20260810 06:20:50.115402 15054 fs_manager.cc:359] Metadata directory not provided
I20260810 06:20:50.115479 15054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:20:50.118928 15054 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:20:50.120757 15071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:50.121824 15054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:20:50.121930 15054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "53cae77a14084637b49c1080fdd82ce7"
format_stamp: "Formatted at 2026-08-10 06:20:49 on dist-test-slave-q9h9"
server_key: "ede8f224f9fe4bf43c16509e68674645"
server_key_iv: "9cc7681dab4d966a66110cf21c6d16b7"
server_key_version: "encryptionkey@0"
I20260810 06:20:50.122421 15054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:20:50.147819 15054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:20:50.150741 15054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:20:50.150898 15054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:20:50.151425 15054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:20:50.152159 15054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:20:50.152205 15054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:50.152264 15054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:20:50.152292 15054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:20:50.160226 15054 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:39665
I20260810 06:20:50.160250 15255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:39665 every 8 connection(s)
I20260810 06:20:50.161098 15054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
Aug 10 06:20:50 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:50.169521 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 15054
Aug 10 06:20:50 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 06:20:50.171175 14843 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:50.162731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:56717 (local address 127.31.204.254:36927)
0810 06:20:50.162855 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:50.162859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:50.163872 (+ 1013us) server_negotiation.cc:407] Connection header received
0810 06:20:50.164682 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:50.164685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:50.164739 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:50.164827 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:50.165874 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.166449 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.167523 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.167739 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.167825 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:50.171059 (+ 3234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.195:56717: BlockingRecv error: recv got EOF from 127.31.204.195:56717 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 06:20:50.171732 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:50.163062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.195:56717)
0810 06:20:50.163744 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:50.163783 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:50.164494 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:50.164992 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:50.164998 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:50.165307 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:50.165735 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:50.165749 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.166570 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.166573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:50.167407 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:50.167417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.167872 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.167876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:50.168023 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:50.168440 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:50.168455 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:50.170956 (+ 2501us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":107,"threads_started":1}
W20260810 06:20:50.172317 15256 heartbeater.cc:644] Failed to heartbeat to 127.31.204.254:36927 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.204.254:36927: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Aug 10 06:20:50 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:50 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:50.177292 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:50.172674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.195:42809)
0810 06:20:50.172783 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:50.172793 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:50.172844 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:50.173106 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:50.173109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:50.173251 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:50.173334 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:50.173339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.173999 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.174002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:50.174552 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:50.174560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.175060 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.175064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:50.175161 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:50.175513 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:50.175525 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:50.177182 (+ 1657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 06:20:50.177368 14843 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:50.172836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:42809 (local address 127.31.204.254:36927)
0810 06:20:50.172946 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:50.172949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:50.172958 (+ 9us) server_negotiation.cc:407] Connection header received
0810 06:20:50.172982 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:50.172984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:50.173013 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:50.173086 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:50.173435 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.173905 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.174659 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.174958 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.175052 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:50.177285 (+ 2233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.195:42809: BlockingRecv error: recv got EOF from 127.31.204.195:42809 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:20:50.179466 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:50.170446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50038 (local address 127.31.204.254:36927)
0810 06:20:50.170646 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:50.170649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:50.171596 (+ 947us) server_negotiation.cc:407] Connection header received
0810 06:20:50.171727 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:50.171729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:50.171768 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:50.171838 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:50.172707 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.173266 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.174133 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.174359 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.174443 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:50.175678 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:20:50.175683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:20:50.175696 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:20:50.175723 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:20:50.177474 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:50.177878 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:50.177884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:50.177889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:50.178074 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:20:50.178362 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:20:50.178364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:20:50.178365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:20:50.178724 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:20:50.178901 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 06:20:50.179126 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 06:20:50.179349 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":69,"threads_started":1}
Aug 10 06:20:50 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:50 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:50.182682 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:50.177829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.195:49951)
0810 06:20:50.177972 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:50.177984 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:50.178078 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:50.178270 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:50.178273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:50.178425 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:50.178491 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:50.178496 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.179370 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.179374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:50.180018 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:50.180027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.180432 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.180435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:50.180508 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:50.180887 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:50.180901 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:50.182569 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260810 06:20:50.182767 14843 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:50.177888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:49951 (local address 127.31.204.254:36927)
0810 06:20:50.178055 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:50.178058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:50.178071 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:20:50.178141 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:50.178143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:50.178180 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:50.178238 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:50.178587 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.179256 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.180142 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.180340 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.180421 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:50.182682 (+ 2261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.195:49951: BlockingRecv error: recv got EOF from 127.31.204.195:49951 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260810 06:20:50.182854 15256 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 06:20:50 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:50 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:50.828579 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:50.823201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.193:41591)
0810 06:20:50.823482 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:50.823494 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:50.823588 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:50.823795 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:50.823797 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:50.823974 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:50.824087 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:50.824092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.824888 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.824891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:50.825583 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:50.825594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.826102 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.826106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:50.826250 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:50.826773 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:50.826785 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:50.828467 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
W20260810 06:20:50.828719 15263 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:50.823277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:41591 (local address 127.31.204.254:36927)
0810 06:20:50.823505 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:50.823509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:50.823556 (+ 47us) server_negotiation.cc:407] Connection header received
0810 06:20:50.823660 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:50.823661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:50.823703 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:50.823782 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:50.824167 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.824757 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.825701 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.825953 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.826096 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:50.828590 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.193:41591: BlockingRecv error: recv got EOF from 127.31.204.193:41591 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
Aug 10 06:20:50 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:50 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:51.000789 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:50.995752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.194:58243)
0810 06:20:50.996010 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:50.996025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:50.996130 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:50.996372 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:50.996375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:50.996568 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:50.996676 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:50.996682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.997411 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.997415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:50.997978 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:50.997986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.998436 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.998440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:50.998533 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:50.999007 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:50.999019 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:51.000674 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
W20260810 06:20:51.000931 15263 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:50.995835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:58243 (local address 127.31.204.254:36927)
0810 06:20:50.995980 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:50.995985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:50.996090 (+ 105us) server_negotiation.cc:407] Connection header received
0810 06:20:50.996218 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:50.996220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:50.996270 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:50.996332 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:50.996789 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.997310 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.998111 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:50.998335 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:50.998422 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:51.000813 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.194:58243: BlockingRecv error: recv got EOF from 127.31.204.194:58243 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 06:20:51 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:51 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:51.188295 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:51.183381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.195:47005)
0810 06:20:51.183670 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:51.183682 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:51.183769 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:51.183961 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:51.183963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:51.184123 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:51.184246 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:51.184252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:51.184923 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:51.184926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:51.185500 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:51.185509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:51.185911 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:51.185915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:51.186043 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:51.186522 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:51.186536 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:51.188192 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":73,"threads_started":1}
W20260810 06:20:51.188436 15263 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:51.183446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:47005 (local address 127.31.204.254:36927)
0810 06:20:51.183571 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:51.183574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:51.183741 (+ 167us) server_negotiation.cc:407] Connection header received
0810 06:20:51.183834 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:51.183836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:51.183877 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:51.183941 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:51.184325 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:51.184833 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:51.185581 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:51.185799 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:51.185859 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:51.188325 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.195:47005: BlockingRecv error: recv got EOF from 127.31.204.195:47005 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 06:20:51 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:51 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:51.834282 15268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:51.829225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.193:35077)
0810 06:20:51.829461 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:51.829473 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:51.829553 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:51.829763 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:51.829765 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:51.829929 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:51.830057 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:51.830065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:51.830765 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:51.830767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:51.831305 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:51.831311 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:51.831661 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:51.831663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:51.831719 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:51.832188 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:51.832200 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:51.834162 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":66,"threads_started":1}
W20260810 06:20:51.834415 15269 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:51.829287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:35077 (local address 127.31.204.254:36927)
0810 06:20:51.829512 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:51.829515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:51.829528 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:20:51.829627 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:51.829630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:51.829670 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:51.829740 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:51.830152 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:51.830675 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:51.831380 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:51.831585 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:51.831646 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:51.834290 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.193:35077: BlockingRecv error: recv got EOF from 127.31.204.193:35077 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":64,"threads_started":1}
Aug 10 06:20:51 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:51 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:52.006578 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:52.001411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.194:41791)
0810 06:20:52.001705 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:52.001720 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:52.001837 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:52.002072 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:52.002075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:52.002287 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:52.002380 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:52.002386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:52.003114 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:52.003117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:52.003692 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:52.003701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:52.004118 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:52.004122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:52.004212 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:52.004646 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:52.004656 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:52.006424 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":75,"threads_started":1}
W20260810 06:20:52.006662 15269 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:52.001553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:41791 (local address 127.31.204.254:36927)
0810 06:20:52.001715 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:52.001720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:52.001792 (+ 72us) server_negotiation.cc:407] Connection header received
0810 06:20:52.001920 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:52.001923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:52.001969 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:52.002051 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:52.002492 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:52.003012 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:52.003804 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:52.004021 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:52.004106 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:52.006562 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.194:41791: BlockingRecv error: recv got EOF from 127.31.204.194:41791 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 06:20:52 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:52 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:52.194123 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:52.188960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.195:50333)
0810 06:20:52.189234 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:52.189249 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:52.189360 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:52.189608 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:52.189610 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:52.189812 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:52.189934 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:52.189943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:52.190735 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:52.190739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:52.191298 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:52.191307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:52.191725 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:52.191729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:52.191815 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:52.192237 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:52.192248 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:52.193932 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":70,"threads_started":1}
W20260810 06:20:52.194352 15269 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:52.189078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:50333 (local address 127.31.204.254:36927)
0810 06:20:52.189195 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:52.189198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:52.189311 (+ 113us) server_negotiation.cc:407] Connection header received
0810 06:20:52.189458 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:52.189460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:52.189509 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:52.189570 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:52.190068 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:52.190622 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:52.191400 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:52.191624 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:52.191711 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:52.194173 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.195:50333: BlockingRecv error: recv got EOF from 127.31.204.195:50333 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 06:20:52 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:52 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:52.845191 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:52.835001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.193:40985)
0810 06:20:52.835267 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:52.835284 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:52.835401 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:52.835773 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:52.835776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:52.836003 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:52.836127 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:52.836135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:52.836898 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:52.836902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:52.837490 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:52.837499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:52.837941 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:52.837945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:52.838054 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:52.842848 (+ 4794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:52.842865 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:52.845031 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":68,"threads_started":1}
W20260810 06:20:52.845311 15275 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:52.835093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:40985 (local address 127.31.204.254:36927)
0810 06:20:52.835356 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:52.835359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:52.835373 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:20:52.835529 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:52.835533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:52.835590 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:52.835692 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:52.836244 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:52.836789 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:52.837603 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:52.837836 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:52.837923 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:52.845192 (+ 7269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.193:40985: BlockingRecv error: recv got EOF from 127.31.204.193:40985 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":96,"threads_started":1}
Aug 10 06:20:53 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:53 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:53.012442 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:53.007270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.194:34829)
0810 06:20:53.007526 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:53.007539 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:53.007663 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:53.007951 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:53.007954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:53.008180 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:53.008314 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:53.008322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:53.009120 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:53.009123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:53.009666 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:53.009675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:53.010094 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:53.010097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:53.010165 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:53.010635 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:53.010649 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:53.012335 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":87,"threads_started":1}
W20260810 06:20:53.012576 15275 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:53.007339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:34829 (local address 127.31.204.254:36927)
0810 06:20:53.007463 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:53.007467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:53.007624 (+ 157us) server_negotiation.cc:407] Connection header received
0810 06:20:53.007750 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:53.007753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:53.007818 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:53.007901 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:53.008428 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:53.009022 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:53.009752 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:53.009970 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:53.010060 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:53.012465 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.194:34829: BlockingRecv error: recv got EOF from 127.31.204.194:34829 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 10 06:20:53 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:53 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:53.199927 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:53.194782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.195:39195)
0810 06:20:53.195067 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:53.195080 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:53.195177 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:53.195406 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:53.195409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:53.195599 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:53.195689 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:53.195695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:53.196506 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:53.196510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:53.197086 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:53.197094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:53.197514 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:53.197517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:53.197610 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:53.198065 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:53.198081 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:53.199806 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":101,"threads_started":1}
W20260810 06:20:53.200042 15275 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:53.194881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:39195 (local address 127.31.204.254:36927)
0810 06:20:53.195006 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:53.195010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:53.195139 (+ 129us) server_negotiation.cc:407] Connection header received
0810 06:20:53.195261 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:53.195264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:53.195312 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:53.195374 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:53.195807 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:53.196402 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:53.197194 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:53.197414 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:53.197500 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:53.199932 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.195:39195: BlockingRecv error: recv got EOF from 127.31.204.195:39195 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 06:20:53 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:53 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:53.851897 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:53.845926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.193:50815)
0810 06:20:53.846230 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:53.846246 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:53.846343 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:53.846610 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:53.846613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:53.846822 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:53.846944 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:53.846952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:53.848043 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:53.848047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:53.848922 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:53.848934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:53.849372 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:53.849375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:53.849435 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:53.849844 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:53.849856 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:53.851752 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":105,"threads_started":1}
W20260810 06:20:53.852011 15280 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:53.846000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:50815 (local address 127.31.204.254:36927)
0810 06:20:53.846312 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:53.846317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:53.846332 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:20:53.846433 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:53.846436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:53.846487 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:53.846575 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:53.847082 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:53.847923 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:53.849046 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:53.849283 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:53.849361 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:53.851898 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.193:50815: BlockingRecv error: recv got EOF from 127.31.204.193:50815 (error 108)
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":113,"threads_started":1}
Aug 10 06:20:54 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:54 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:54.017773 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:54.013065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.194:46751)
0810 06:20:54.013331 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:54.013344 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:54.013434 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:54.013628 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:54.013630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:54.013798 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:54.013890 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:54.013894 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:54.014588 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:54.014590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:54.015146 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:54.015155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:54.015506 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:54.015509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:54.015572 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:54.015999 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:54.016010 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:54.017659 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":87,"threads_started":1}
W20260810 06:20:54.017887 15280 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:54.013140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:46751 (local address 127.31.204.254:36927)
0810 06:20:54.013304 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:54.013307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:54.013403 (+ 96us) server_negotiation.cc:407] Connection header received
0810 06:20:54.013500 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:54.013502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:54.013544 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:54.013608 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:54.013968 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:54.014509 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:54.015229 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:54.015429 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:54.015496 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:54.017788 (+ 2292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.194:46751: BlockingRecv error: recv got EOF from 127.31.204.194:46751 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 06:20:54 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:54 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:54.205416 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:54.200564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.195:58057)
0810 06:20:54.200843 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:54.200855 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:54.200946 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:54.201143 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:54.201145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:54.201311 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:54.201400 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:54.201405 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:54.202101 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:54.202103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:54.202637 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:54.202645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:54.203031 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:54.203034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:54.203091 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:54.203499 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:54.203517 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:54.205274 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":94,"threads_started":1}
W20260810 06:20:54.205504 15280 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:54.200647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:58057 (local address 127.31.204.254:36927)
0810 06:20:54.200789 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:54.200795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:54.200946 (+ 151us) server_negotiation.cc:407] Connection header received
0810 06:20:54.201007 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:54.201009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:54.201055 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:54.201122 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:54.201496 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:54.202034 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:54.202730 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:54.202949 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:54.203024 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:54.205407 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.195:58057: BlockingRecv error: recv got EOF from 127.31.204.195:58057 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 06:20:54 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:54 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:54.857730 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:54.852623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.193:43605)
0810 06:20:54.852886 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:54.852898 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:54.852982 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:54.853197 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:54.853199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:54.853384 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:54.853478 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:54.853483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:54.854238 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:54.854240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:54.854819 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:54.854827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:54.855255 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:54.855257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:54.855324 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:54.855751 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:54.855763 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:54.857587 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":84,"threads_started":1}
W20260810 06:20:54.857880 15288 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:54.852690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:43605 (local address 127.31.204.254:36927)
0810 06:20:54.852948 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:54.852951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:54.852965 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:20:54.853046 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:54.853048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:54.853103 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:54.853175 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:54.853558 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:54.854153 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:54.854922 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:54.855169 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:54.855249 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:54.857747 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.193:43605: BlockingRecv error: recv got EOF from 127.31.204.193:43605 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":59,"threads_started":1}
Aug 10 06:20:55 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:55 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:55.023556 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:55.018437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.194:35003)
0810 06:20:55.018690 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:55.018704 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:55.018816 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:55.019045 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:55.019048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:55.019238 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:55.019328 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:55.019333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:55.020049 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:55.020053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:55.020616 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:55.020625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:55.021041 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:55.021045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:55.021128 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:55.021545 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:55.021556 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:55.023417 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
W20260810 06:20:55.023631 15288 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:55.018565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:35003 (local address 127.31.204.254:36927)
0810 06:20:55.018693 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:55.018697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:55.018755 (+ 58us) server_negotiation.cc:407] Connection header received
0810 06:20:55.018898 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:55.018901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:55.018947 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:55.019010 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:55.019431 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:55.019949 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:55.020721 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:55.020944 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:55.021028 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:55.023535 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.194:35003: BlockingRecv error: recv got EOF from 127.31.204.194:35003 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 06:20:55 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:55 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:55.211229 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:55.206098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.195:60065)
0810 06:20:55.206398 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:55.206412 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:55.206515 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:55.206762 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:55.206765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:55.206964 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:55.207066 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:55.207072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:55.207838 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:55.207842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:55.208426 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:55.208435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:55.208880 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:55.208884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:55.208972 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:55.209419 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:55.209430 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:55.211067 (+ 1637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":111,"threads_started":1}
W20260810 06:20:55.211304 15288 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:55.206169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:60065 (local address 127.31.204.254:36927)
0810 06:20:55.206334 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:55.206339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:55.206481 (+ 142us) server_negotiation.cc:407] Connection header received
0810 06:20:55.206594 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:55.206607 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:55.206658 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:55.206721 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:55.207197 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:55.207728 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:55.208540 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:55.208777 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:55.208863 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:55.211206 (+ 2343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.195:60065: BlockingRecv error: recv got EOF from 127.31.204.195:60065 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 06:20:55 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:55 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:55.863528 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:55.858456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.193:57667)
0810 06:20:55.858706 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:55.858720 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:55.858813 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:55.859017 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:55.859019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:55.859204 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:55.859301 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:55.859306 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:55.859983 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:55.859985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:55.860681 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:55.860695 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:55.861228 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:55.861231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:55.861292 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:55.861777 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:55.861789 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:55.863417 (+ 1628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":68,"threads_started":1}
W20260810 06:20:55.863651 15294 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:55.858511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:57667 (local address 127.31.204.254:36927)
0810 06:20:55.858726 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:55.858729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:55.858777 (+ 48us) server_negotiation.cc:407] Connection header received
0810 06:20:55.858878 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:55.858880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:55.858921 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:55.858996 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:55.859381 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:55.859903 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:55.860792 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:55.861108 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:55.861184 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:55.863540 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.193:57667: BlockingRecv error: recv got EOF from 127.31.204.193:57667 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":66,"threads_started":1}
Aug 10 06:20:56 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:56 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:56.029604 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:56.024364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.194:48111)
0810 06:20:56.024627 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:56.024639 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:56.024732 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:56.024914 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:56.024916 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:56.025092 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:56.025203 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:56.025210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:56.025931 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:56.025933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:56.026574 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:56.026585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:56.026961 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:56.026963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:56.027018 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:56.027536 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:56.027550 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:56.029463 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
W20260810 06:20:56.029713 15294 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:56.024437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:48111 (local address 127.31.204.254:36927)
0810 06:20:56.024566 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:56.024570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:56.024696 (+ 126us) server_negotiation.cc:407] Connection header received
0810 06:20:56.024796 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:56.024798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:56.024839 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:56.024907 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:56.025293 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:56.025840 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:56.026668 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:56.026880 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:56.026953 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:56.029612 (+ 2659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.194:48111: BlockingRecv error: recv got EOF from 127.31.204.194:48111 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 06:20:56 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:56 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:56.217032 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:56.211946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.195:34101)
0810 06:20:56.212229 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:56.212243 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:56.212339 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:56.212588 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:56.212592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:56.212774 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:56.212873 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:56.212878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:56.213593 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:56.213596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:56.214209 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:56.214218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:56.214614 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:56.214617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:56.214680 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:56.215134 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:56.215144 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:56.216918 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":73,"threads_started":1}
W20260810 06:20:56.217159 15294 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:56.212018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:34101 (local address 127.31.204.254:36927)
0810 06:20:56.212152 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:56.212157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:56.212306 (+ 149us) server_negotiation.cc:407] Connection header received
0810 06:20:56.212406 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:56.212408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:56.212452 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:56.212516 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:56.212965 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:56.213479 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:56.214306 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:56.214523 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:56.214597 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:56.217045 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.195:34101: BlockingRecv error: recv got EOF from 127.31.204.195:34101 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 06:20:56 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:56 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:56.869843 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:56.864227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.193:41165)
0810 06:20:56.864540 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:56.864553 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:56.864662 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:56.864869 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:56.864871 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:56.865052 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:56.865164 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:56.865170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:56.865999 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:56.866003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:56.866795 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:56.866808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:56.867289 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:56.867293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:56.867372 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:56.867864 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:56.867878 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:56.869725 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":71,"threads_started":1}
W20260810 06:20:56.870033 15299 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:56.864300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:41165 (local address 127.31.204.254:36927)
0810 06:20:56.864555 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:56.864559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:56.864608 (+ 49us) server_negotiation.cc:407] Connection header received
0810 06:20:56.864728 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:56.864730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:56.864773 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:56.864845 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:56.865247 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:56.865846 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:56.866908 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:56.867177 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:56.867271 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:56.869873 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.193:41165: BlockingRecv error: recv got EOF from 127.31.204.193:41165 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
Aug 10 06:20:57 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:57 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:57.035495 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:57.030350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.194:47867)
0810 06:20:57.030632 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:57.030646 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:57.030745 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:57.030943 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:57.030945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:57.031123 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:57.031233 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:57.031239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:57.031899 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:57.031901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:57.032449 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:57.032455 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:57.032804 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:57.032807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:57.032862 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:57.033303 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:57.033314 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:57.035351 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":75,"threads_started":1}
W20260810 06:20:57.035589 15299 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:57.030423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:47867 (local address 127.31.204.254:36927)
0810 06:20:57.030588 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:57.030593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:57.030715 (+ 122us) server_negotiation.cc:407] Connection header received
0810 06:20:57.030812 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:57.030814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:57.030858 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:57.030921 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:57.031315 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:57.031830 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:57.032525 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:57.032733 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:57.032795 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:57.035491 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.194:47867: BlockingRecv error: recv got EOF from 127.31.204.194:47867 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 06:20:57 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:57 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:57.222564 15302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:57.217726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.195:59475)
0810 06:20:57.218068 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:57.218081 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:57.218175 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:57.218364 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:57.218366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:57.218538 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:57.218629 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:57.218634 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:57.219299 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:57.219301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:57.219836 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:57.219842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:57.220189 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:57.220191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:57.220247 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:57.220679 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:57.220690 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:57.222420 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":88,"threads_started":1}
W20260810 06:20:57.222651 15299 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:57.217787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:59475 (local address 127.31.204.254:36927)
0810 06:20:57.217919 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:57.217922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:57.218137 (+ 215us) server_negotiation.cc:407] Connection header received
0810 06:20:57.218248 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:57.218250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:57.218296 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:57.218349 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:57.218708 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:57.219218 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:57.219913 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:57.220114 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:57.220179 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:57.222553 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.195:59475: BlockingRecv error: recv got EOF from 127.31.204.195:59475 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 06:20:57 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:57 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:57.877918 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:57.870489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.193:33207)
0810 06:20:57.870754 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:57.870767 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:57.870855 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:57.871059 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:57.871061 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:57.871227 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:57.871324 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:57.871329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:57.871995 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:57.871997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:57.872540 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:57.872545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:57.872898 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:57.872901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:57.872957 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:57.876079 (+ 3122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:57.876091 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:57.877770 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":82,"threads_started":1}
W20260810 06:20:57.878041 15304 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:57.870601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:33207 (local address 127.31.204.254:36927)
0810 06:20:57.870819 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:57.870822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:57.870835 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:20:57.870925 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:57.870927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:57.870968 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:57.871036 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:57.871401 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:57.871926 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:57.872612 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:57.872826 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:57.872889 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:57.877901 (+ 5012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.193:33207: BlockingRecv error: recv got EOF from 127.31.204.193:33207 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":53,"threads_started":1}
Aug 10 06:20:58 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:58 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:58.041219 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:58.036223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.194:55397)
0810 06:20:58.036497 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:58.036509 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:58.036607 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:58.036806 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:58.036808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:58.036984 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:58.037079 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:58.037084 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:58.037885 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:58.037887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:58.038455 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:58.038462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:58.038831 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:58.038833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:58.038887 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:58.039332 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:58.039343 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:58.041074 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":89,"threads_started":1}
W20260810 06:20:58.041405 15304 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:58.036287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:55397 (local address 127.31.204.254:36927)
0810 06:20:58.036461 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:58.036466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:58.036572 (+ 106us) server_negotiation.cc:407] Connection header received
0810 06:20:58.036673 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:58.036675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:58.036720 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:58.036784 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:58.037157 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:58.037812 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:58.038538 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:58.038752 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:58.038822 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:58.041212 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.194:55397: BlockingRecv error: recv got EOF from 127.31.204.194:55397 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 10 06:20:58 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:58 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:58.228569 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:58.223348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.195:48359)
0810 06:20:58.223610 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:58.223622 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:58.223717 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:58.223977 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:58.223979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:58.224156 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:58.224254 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:58.224271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:58.224939 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:58.224942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:58.225522 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:58.225530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:58.225916 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:58.225918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:58.225982 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:58.226546 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:58.226560 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:58.228456 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":88,"threads_started":1}
W20260810 06:20:58.228749 15304 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:58.223441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:48359 (local address 127.31.204.254:36927)
0810 06:20:58.223575 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:58.223578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:58.223681 (+ 103us) server_negotiation.cc:407] Connection header received
0810 06:20:58.223823 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:58.223826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:58.223878 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:58.223950 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:58.224348 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:58.224864 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:58.225615 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:58.225832 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:58.225903 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:58.228623 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.195:48359: BlockingRecv error: recv got EOF from 127.31.204.195:48359 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 06:20:58 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:58 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:58.883697 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:58.878586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.193:36563)
0810 06:20:58.878862 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:58.878877 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:58.878974 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:58.879222 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:58.879225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:58.879404 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:58.879510 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:58.879516 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:58.880248 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:58.880251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:58.880823 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:58.880832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:58.881211 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:58.881213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:58.881269 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:58.881698 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:58.881708 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:58.883573 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":84,"threads_started":1}
W20260810 06:20:58.883832 15313 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:58.878644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:36563 (local address 127.31.204.254:36927)
0810 06:20:58.878885 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:58.878889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:58.878934 (+ 45us) server_negotiation.cc:407] Connection header received
0810 06:20:58.879059 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:58.879062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:58.879113 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:58.879194 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:58.879597 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:58.880154 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:58.880909 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:58.881123 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:58.881189 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:58.883697 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.193:36563: BlockingRecv error: recv got EOF from 127.31.204.193:36563 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
Aug 10 06:20:59 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:59 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:59.047205 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:59.041933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.194:39879)
0810 06:20:59.042225 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:59.042239 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:59.042336 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:59.042530 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:59.042532 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:59.042714 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:59.042812 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:59.042817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:59.043517 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:59.043521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:59.044279 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:59.044292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:59.044758 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:59.044761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:59.044822 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:59.045271 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:59.045282 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:59.047052 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":106,"threads_started":1}
W20260810 06:20:59.047307 15313 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:59.042034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:39879 (local address 127.31.204.254:36927)
0810 06:20:59.042193 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:59.042197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:59.042299 (+ 102us) server_negotiation.cc:407] Connection header received
0810 06:20:59.042407 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:59.042409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:59.042452 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:59.042504 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:59.042893 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:59.043422 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:59.044395 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:59.044656 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:59.044734 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:59.047179 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.194:39879: BlockingRecv error: recv got EOF from 127.31.204.194:39879 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 06:20:59 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:59 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:59.234308 15317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:59.229238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.195:43473)
0810 06:20:59.229483 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:59.229496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:59.229592 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:59.229792 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:59.229794 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:59.229969 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:59.230098 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:59.230105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:59.230774 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:59.230776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:59.231324 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:59.231332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:59.231688 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:59.231690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:59.231748 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:59.232180 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:59.232191 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:59.234191 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
W20260810 06:20:59.234385 15313 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:59.229301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:43473 (local address 127.31.204.254:36927)
0810 06:20:59.229428 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:59.229431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:59.229554 (+ 123us) server_negotiation.cc:407] Connection header received
0810 06:20:59.229658 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:59.229660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:59.229704 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:59.229776 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:59.230185 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:59.230686 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:59.231408 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:59.231612 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:59.231677 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:59.234294 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.195:43473: BlockingRecv error: recv got EOF from 127.31.204.195:43473 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 06:20:59 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:20:59 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:20:59.889339 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:20:59.884368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.193:53669)
0810 06:20:59.884634 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 06:20:59.884647 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:20:59.884737 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:20:59.884949 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:20:59.884951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:20:59.885122 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:20:59.885217 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:59.885222 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:59.885889 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:59.885891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:59.886480 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:20:59.886490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:20:59.886872 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:20:59.886874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:20:59.886932 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:59.887393 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:20:59.887405 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:20:59.889180 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
W20260810 06:20:59.889432 15324 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:20:59.884431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:53669 (local address 127.31.204.254:36927)
0810 06:20:59.884648 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 06:20:59.884651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:20:59.884709 (+ 58us) server_negotiation.cc:407] Connection header received
0810 06:20:59.884812 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:20:59.884814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:20:59.884853 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:20:59.884926 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:20:59.885296 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:59.885811 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:59.886570 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:20:59.886790 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:20:59.886863 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:20:59.889320 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.193:53669: BlockingRecv error: recv got EOF from 127.31.204.193:53669 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":65,"threads_started":1}
Aug 10 06:21:00 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:21:00 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:21:00.052973 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:00.047963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.194:60947)
0810 06:21:00.048236 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:00.048252 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:00.048360 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:00.048600 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:00.048603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:00.048787 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:00.048888 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:00.048893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:00.049566 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:00.049568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:00.050150 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:00.050159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:00.050580 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:00.050582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:00.050646 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:00.051085 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:00.051096 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:00.052846 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":71,"threads_started":1}
W20260810 06:21:00.053079 15324 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:21:00.048012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:60947 (local address 127.31.204.254:36927)
0810 06:21:00.048144 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:00.048147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:00.048311 (+ 164us) server_negotiation.cc:407] Connection header received
0810 06:21:00.048444 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:00.048447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:00.048489 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:00.048552 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:00.048975 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:00.049491 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:00.050241 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:00.050497 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:00.050572 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:00.052969 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.194:60947: BlockingRecv error: recv got EOF from 127.31.204.194:60947 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 10 06:21:00 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:21:00 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:21:00.240319 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:00.235046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.195:42313)
0810 06:21:00.235354 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:00.235369 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:00.235475 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:00.235740 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:00.235743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:00.235966 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:00.236073 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:00.236089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:00.236846 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:00.236850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:00.237418 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:00.237427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:00.237876 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:00.237880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:00.237963 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:00.238443 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:00.238454 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:00.240177 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":103,"threads_started":1}
W20260810 06:21:00.240406 15324 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:21:00.235199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:42313 (local address 127.31.204.254:36927)
0810 06:21:00.235338 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:00.235342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:00.235439 (+ 97us) server_negotiation.cc:407] Connection header received
0810 06:21:00.235559 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:00.235561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:00.235617 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:00.235688 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:00.236193 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:00.236743 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:00.237539 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:00.237777 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:00.237862 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:00.240305 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.195:42313: BlockingRecv error: recv got EOF from 127.31.204.195:42313 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 06:21:00 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:21:00 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:21:00.895232 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:00.890087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.193:38875)
0810 06:21:00.890404 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:00.890419 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:00.890529 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:00.890772 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:00.890775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:00.890977 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:00.891072 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:00.891078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:00.891819 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:00.891823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:00.892376 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:00.892384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:00.892811 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:00.892815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:00.892901 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:00.893344 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:00.893354 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:00.895105 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":109,"threads_started":1}
W20260810 06:21:00.895363 15330 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:21:00.890121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:38875 (local address 127.31.204.254:36927)
0810 06:21:00.890385 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:00.890390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:00.890483 (+ 93us) server_negotiation.cc:407] Connection header received
0810 06:21:00.890610 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:00.890613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:00.890662 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:00.890736 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:00.891183 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:00.891716 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:00.892483 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:00.892711 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:00.892803 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:00.895248 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.193:38875: BlockingRecv error: recv got EOF from 127.31.204.193:38875 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
Aug 10 06:21:01 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:21:01 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:21:01.058439 15332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:01.053646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.194:51219)
0810 06:21:01.053889 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:01.053902 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:01.053998 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:01.054206 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:01.054208 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:01.054393 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:01.054490 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:01.054495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:01.055175 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:01.055177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:01.055717 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:01.055723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:01.056116 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:01.056118 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:01.056171 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:01.056606 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:01.056616 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:01.058326 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":95,"threads_started":1}
W20260810 06:21:01.058548 15330 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:21:01.053701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:51219 (local address 127.31.204.254:36927)
0810 06:21:01.053817 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:01.053820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:01.053951 (+ 131us) server_negotiation.cc:407] Connection header received
0810 06:21:01.054084 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:01.054086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:01.054125 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:01.054180 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:01.054572 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:01.055100 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:01.055812 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:01.056033 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:01.056110 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:01.058450 (+ 2340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.194:51219: BlockingRecv error: recv got EOF from 127.31.204.194:51219 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 10 06:21:01 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:21:01 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:21:01.246110 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:01.241033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.195:34699)
0810 06:21:01.241278 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:01.241291 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:01.241397 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:01.241677 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:01.241680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:01.241922 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:01.242069 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:01.242079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:01.242882 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:01.242885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:01.243431 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:01.243440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:01.243789 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:01.243791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:01.243849 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:01.244297 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:01.244307 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:01.245963 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":77,"threads_started":1}
W20260810 06:21:01.246210 15330 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:21:01.241096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:34699 (local address 127.31.204.254:36927)
0810 06:21:01.241237 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:01.241242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:01.241364 (+ 122us) server_negotiation.cc:407] Connection header received
0810 06:21:01.241482 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:01.241485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:01.241540 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:01.241613 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:01.242181 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:01.242786 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:01.243516 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:01.243717 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:01.243780 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:01.246114 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.195:34699: BlockingRecv error: recv got EOF from 127.31.204.195:34699 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 06:21:01 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:21:01 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:21:01.900854 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:01.895921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.193:37323)
0810 06:21:01.896171 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:01.896184 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:01.896292 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:01.896498 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:01.896500 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:01.896677 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:01.896776 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:01.896781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:01.897449 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:01.897451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:01.897991 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:01.897997 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:01.898466 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:01.898469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:01.898531 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:01.899000 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:01.899011 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:01.900704 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":67,"threads_started":1}
W20260810 06:21:01.900964 15337 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:21:01.895978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:37323 (local address 127.31.204.254:36927)
0810 06:21:01.896200 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:01.896203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:01.896253 (+ 50us) server_negotiation.cc:407] Connection header received
0810 06:21:01.896358 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:01.896360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:01.896402 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:01.896476 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:01.896857 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:01.897374 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:01.898123 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:01.898369 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:01.898442 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:01.900853 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.193:37323: BlockingRecv error: recv got EOF from 127.31.204.193:37323 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":69,"threads_started":1}
Aug 10 06:21:02 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:21:02 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:21:02.064229 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:02.059089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.194:55983)
0810 06:21:02.059362 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:02.059374 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:02.059480 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:02.059709 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:02.059712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:02.059905 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:02.059999 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:02.060006 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:02.060727 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:02.060731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:02.061298 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:02.061307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:02.061783 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:02.061791 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:02.061875 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:02.062401 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:02.062416 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:02.064113 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":89,"threads_started":1}
W20260810 06:21:02.064347 15337 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:21:02.059163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:55983 (local address 127.31.204.254:36927)
0810 06:21:02.059313 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:02.059317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:02.059442 (+ 125us) server_negotiation.cc:407] Connection header received
0810 06:21:02.059564 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:02.059567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:02.059613 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:02.059676 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:02.060108 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:02.060626 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:02.061406 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:02.061678 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:02.061774 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:02.064236 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.194:55983: BlockingRecv error: recv got EOF from 127.31.204.194:55983 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 06:21:02 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:21:02 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:21:02.251925 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:02.246831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.195:38889)
0810 06:21:02.247111 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:02.247124 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:02.247219 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:02.247432 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:02.247434 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:02.247680 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:02.247782 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:02.247788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:02.248482 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:02.248485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:02.249048 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:02.249057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:02.249448 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:02.249450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:02.249508 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:02.249998 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:02.250010 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:02.251774 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":109,"threads_started":1}
W20260810 06:21:02.252017 15337 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:21:02.246902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:38889 (local address 127.31.204.254:36927)
0810 06:21:02.247030 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:02.247033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:02.247180 (+ 147us) server_negotiation.cc:407] Connection header received
0810 06:21:02.247306 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:02.247308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:02.247353 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:02.247408 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:02.247884 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:02.248409 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:02.249146 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:02.249367 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:02.249438 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:02.251905 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.195:38889: BlockingRecv error: recv got EOF from 127.31.204.195:38889 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 06:21:02 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:21:02 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:21:02.910544 15342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:02.901681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.193:58061)
0810 06:21:02.901962 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:02.901976 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:02.902120 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:02.902345 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:02.902347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:02.902543 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:02.902654 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:02.902659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:02.903362 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:02.903364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:02.903929 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:02.903938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:02.904323 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:02.904326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:02.904388 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:02.908655 (+ 4267us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:02.908668 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:02.910419 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":88,"threads_started":1}
W20260810 06:21:02.910665 15343 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:21:02.901708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:58061 (local address 127.31.204.254:36927)
0810 06:21:02.901970 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:02.901974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:02.902059 (+ 85us) server_negotiation.cc:407] Connection header received
0810 06:21:02.902201 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:02.902203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:02.902248 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:02.902328 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:02.902748 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:02.903274 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:02.904020 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:02.904235 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:02.904309 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:02.910541 (+ 6232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.193:58061: BlockingRecv error: recv got EOF from 127.31.204.193:58061 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
Aug 10 06:21:03 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:21:03 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:21:03.070216 15344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:03.064885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.194:52043)
0810 06:21:03.065135 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:03.065149 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:03.065256 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:03.065499 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:03.065502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:03.065689 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:03.065785 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:03.065791 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:03.066582 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:03.066586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:03.067176 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:03.067187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:03.067738 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:03.067741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:03.067829 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:03.068314 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:03.068325 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:03.070057 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":78,"threads_started":1}
W20260810 06:21:03.070464 15343 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:21:03.065007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:52043 (local address 127.31.204.254:36927)
0810 06:21:03.065137 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:03.065141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:03.065209 (+ 68us) server_negotiation.cc:407] Connection header received
0810 06:21:03.065349 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:03.065352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:03.065400 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:03.065467 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:03.065891 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:03.066478 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:03.067304 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:03.067628 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:03.067736 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:03.070270 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.194:52043: BlockingRecv error: recv got EOF from 127.31.204.194:52043 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 06:21:03 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:21:03 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:21:03.257767 15345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:03.252554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.195:54067)
0810 06:21:03.252855 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:03.252867 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:03.252970 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:03.253205 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:03.253209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:03.253411 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:03.253505 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:03.253512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:03.254256 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:03.254259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:03.254826 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:03.254834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:03.255245 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:03.255249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:03.255332 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:03.255776 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:03.255787 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:03.257644 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":116,"threads_started":1}
W20260810 06:21:03.257889 15343 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:21:03.252624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:54067 (local address 127.31.204.254:36927)
0810 06:21:03.252747 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:03.252751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:03.252931 (+ 180us) server_negotiation.cc:407] Connection header received
0810 06:21:03.253056 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:03.253059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:03.253105 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:03.253168 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:03.253612 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:03.254161 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:03.254931 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:03.255148 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:03.255230 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:03.257776 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.195:54067: BlockingRecv error: recv got EOF from 127.31.204.195:54067 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 06:21:03 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:21:03 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:21:03.916486 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:03.911278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.193:57757)
0810 06:21:03.911549 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:03.911563 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:03.911661 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:03.911873 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:03.911875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:03.912052 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:03.912152 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:03.912157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:03.912840 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:03.912842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:03.913591 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:03.913604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:03.914139 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:03.914142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:03.914203 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:03.914648 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:03.914659 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:03.916336 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
W20260810 06:21:03.916584 15348 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:21:03.911375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:57757 (local address 127.31.204.254:36927)
0810 06:21:03.911679 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:03.911682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:03.911694 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:21:03.911726 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:03.911728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:03.911775 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:03.911848 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:03.912232 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:03.912748 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:03.913704 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:03.914040 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:03.914115 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:03.916471 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.193:57757: BlockingRecv error: recv got EOF from 127.31.204.193:57757 (error 108)
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":63,"threads_started":1}
Aug 10 06:21:04 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:21:04 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:21:04.075814 15349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:04.070902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.194:57657)
0810 06:21:04.071155 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:04.071167 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:04.071280 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:04.071517 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:04.071520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:04.071715 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:04.071806 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:04.071812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:04.072528 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:04.072532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:04.073075 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:04.073083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:04.073505 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:04.073509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:04.073592 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:04.074034 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:04.074045 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:04.075673 (+ 1628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
W20260810 06:21:04.075922 15348 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:21:04.070965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:57657 (local address 127.31.204.254:36927)
0810 06:21:04.071107 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:04.071112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:04.071234 (+ 122us) server_negotiation.cc:407] Connection header received
0810 06:21:04.071367 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:04.071370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:04.071415 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:04.071478 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:04.071918 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:04.072430 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:04.073186 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:04.073407 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:04.073493 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:04.075824 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.194:57657: BlockingRecv error: recv got EOF from 127.31.204.194:57657 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 06:21:04 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:21:04 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.195@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:21:04.263690 15350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:04.258473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.195:60091)
0810 06:21:04.258789 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:04.258804 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:04.258923 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:04.259202 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:04.259205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:04.259442 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:04.259563 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:04.259572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:04.260390 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:04.260393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:04.260944 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:04.260953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:04.261317 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:04.261319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:04.261376 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:04.261828 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:04.261839 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:04.263577 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":92,"threads_started":1}
W20260810 06:21:04.263836 15348 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:21:04.258574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:60091 (local address 127.31.204.254:36927)
0810 06:21:04.258746 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:04.258750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:04.258870 (+ 120us) server_negotiation.cc:407] Connection header received
0810 06:21:04.259016 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:04.259019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:04.259072 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:04.259168 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:04.259705 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:04.260292 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:04.261031 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:04.261243 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:04.261308 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:04.263720 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.195:60091: BlockingRecv error: recv got EOF from 127.31.204.195:60091 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 06:21:04 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:21:04 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.193@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:21:04.922520 15353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:04.917411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.193:56885)
0810 06:21:04.917682 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:04.917703 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:04.917795 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:04.918003 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:04.918005 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:04.918207 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:04.918303 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:04.918308 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:04.919106 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:04.919110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:04.919784 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:04.919796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:04.920202 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:04.920205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:04.920261 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:04.920710 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:04.920724 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:04.922407 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
W20260810 06:21:04.922644 15354 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:21:04.917492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:56885 (local address 127.31.204.254:36927)
0810 06:21:04.917707 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:04.917710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:04.917772 (+ 62us) server_negotiation.cc:407] Connection header received
0810 06:21:04.917862 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:04.917864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:04.917907 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:04.917978 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:04.918411 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:04.918998 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:04.919890 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:04.920117 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:04.920193 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:04.922534 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.193:56885: BlockingRecv error: recv got EOF from 127.31.204.193:56885 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":57,"threads_started":1}
Aug 10 06:21:05 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:21:05 dist-test-slave-q9h9 krb5kdc[13665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.204.194@KRBTEST.COM for oryx/127.31.204.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:21:05.081712 15357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:05.076618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36927 (local address 127.31.204.194:40893)
0810 06:21:05.076891 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:05.076903 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:05.077003 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:05.077227 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:05.077230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:05.077431 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:05.077525 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:05.077531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:05.078300 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:05.078303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:05.078850 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:05.078858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:05.079300 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:05.079304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:05.079384 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:05.079858 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:05.079871 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:05.081554 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.204.254:36927: Server oryx/127.31.204.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
W20260810 06:21:05.081785 15354 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:21:05.076698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:40893 (local address 127.31.204.254:36927)
0810 06:21:05.076846 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:05.076851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:05.076963 (+ 112us) server_negotiation.cc:407] Connection header received
0810 06:21:05.077083 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:05.077086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:05.077132 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:05.077196 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:05.077632 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:05.078193 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:05.078953 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:05.079202 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:05.079292 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:05.081687 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.204.194:40893: BlockingRecv error: recv got EOF from 127.31.204.194:40893 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 06:21:05.175335 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 14615
I20260810 06:21:05.190223 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 14846
I20260810 06:21:05.201736 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 15054
I20260810 06:21:05.218568 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 14504
2026-08-10T06:21:05Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17981 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T06:21:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:21:05Z Disabled control of system clock
I20260810 06:21:05.252321 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:46101
--webserver_interface=127.31.204.254
--webserver_port=0
--builtin_ntp_servers=127.31.204.212:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:46101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 06:21:05.359087 15363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:21:05.359309 15363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:21:05.359341 15363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:21:05.363104 15363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:21:05.363152 15363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:21:05.363165 15363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:21:05.363183 15363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:21:05.363202 15363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:21:05.367669 15363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:46101
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:46101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:21:05.368494 15363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:21:05.369660 15363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:21:05.377463 15363 server_base.cc:1034] running on GCE node
W20260810 06:21:05.377552 15368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:05.377630 15371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:05.377805 15369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:05.378322 15363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:21:05.379276 15363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:21:05.380425 15363 hybrid_clock.cc:648] HybridClock initialized: now 1786342865380408 us; error 34 us; skew 500 ppm
I20260810 06:21:05.382004 15363 webserver.cc:460] Webserver started at http://127.31.204.254:44445/ using document root <none> and password file <none>
I20260810 06:21:05.382521 15363 fs_manager.cc:359] Metadata directory not provided
I20260810 06:21:05.382586 15363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:21:05.382768 15363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:21:05.384593 15363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "203da7427b9940159752bdbd7fc68a7f"
format_stamp: "Formatted at 2026-08-10 06:21:05 on dist-test-slave-q9h9"
server_key: "b34cc590dabd643c6e57737b261d6291"
server_key_iv: "f49f27cacb90753a626fcdfefb2c9322"
server_key_version: "encryptionkey@0"
I20260810 06:21:05.385105 15363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "203da7427b9940159752bdbd7fc68a7f"
format_stamp: "Formatted at 2026-08-10 06:21:05 on dist-test-slave-q9h9"
server_key: "b34cc590dabd643c6e57737b261d6291"
server_key_iv: "f49f27cacb90753a626fcdfefb2c9322"
server_key_version: "encryptionkey@0"
I20260810 06:21:05.388437 15363 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260810 06:21:05.390544 15388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:05.391609 15363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:21:05.391726 15363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "203da7427b9940159752bdbd7fc68a7f"
format_stamp: "Formatted at 2026-08-10 06:21:05 on dist-test-slave-q9h9"
server_key: "b34cc590dabd643c6e57737b261d6291"
server_key_iv: "f49f27cacb90753a626fcdfefb2c9322"
server_key_version: "encryptionkey@0"
I20260810 06:21:05.391814 15363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:21:05.408674 15363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:21:05.409247 15363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:21:05.409396 15363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:21:05.415660 15363 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:46101
I20260810 06:21:05.415675 15474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:46101 every 8 connection(s)
I20260810 06:21:05.416515 15363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:21:05.417506 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 15363
I20260810 06:21:05.417624 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:21:05.417927 32563 external_mini_cluster.cc:1428] Setting key 9966efbaf0974e16447d59510c3748bb
I20260810 06:21:05.419956 15475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:21:05.425037 15475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f: Bootstrap starting.
I20260810 06:21:05.426429 15481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:05.419282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53226 (local address 127.31.204.254:46101)
0810 06:21:05.419896 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:05.419910 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:05.419943 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:21:05.420568 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:05.420582 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:05.420943 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:05.421235 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:05.421612 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:05.422597 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:05.423453 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:05.423887 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:05.424025 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:05.424830 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:21:05.424834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:21:05.424846 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:21:05.425215 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:21:05.425270 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:21:05.425289 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:21:05.425371 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:05.425514 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 06:21:05.425745 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":138,"threads_started":1}
I20260810 06:21:05.427136 15475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f: Neither blocks nor log segments found. Creating new log.
I20260810 06:21:05.427812 15475 log.cc:824] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f: Log is configured to *not* fsync() on all Append() calls
I20260810 06:21:05.429400 15475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f: No bootstrap required, opened a new log
I20260810 06:21:05.432500 15475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203da7427b9940159752bdbd7fc68a7f" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46101 } }
I20260810 06:21:05.432690 15475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:21:05.432755 15475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 203da7427b9940159752bdbd7fc68a7f, State: Initialized, Role: FOLLOWER
I20260810 06:21:05.433127 15475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203da7427b9940159752bdbd7fc68a7f" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46101 } }
I20260810 06:21:05.433248 15475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:21:05.433319 15475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:21:05.433413 15475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:21:05.434192 15475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203da7427b9940159752bdbd7fc68a7f" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46101 } }
I20260810 06:21:05.434478 15475 leader_election.cc:302] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 203da7427b9940159752bdbd7fc68a7f; no voters:
I20260810 06:21:05.434729 15475 leader_election.cc:288] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:21:05.434834 15494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:21:05.435026 15494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f [term 1 LEADER]: Becoming Leader. State: Replica: 203da7427b9940159752bdbd7fc68a7f, State: Running, Role: LEADER
I20260810 06:21:05.435338 15494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203da7427b9940159752bdbd7fc68a7f" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46101 } }
I20260810 06:21:05.435778 15475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:21:05.436767 15497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 203da7427b9940159752bdbd7fc68a7f. Latest consensus state: current_term: 1 leader_uuid: "203da7427b9940159752bdbd7fc68a7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203da7427b9940159752bdbd7fc68a7f" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46101 } } }
I20260810 06:21:05.436887 15497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f [sys.catalog]: This master's current role is: LEADER
I20260810 06:21:05.437269 15508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:21:05.437194 15496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "203da7427b9940159752bdbd7fc68a7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203da7427b9940159752bdbd7fc68a7f" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46101 } } }
I20260810 06:21:05.438228 15496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f [sys.catalog]: This master's current role is: LEADER
I20260810 06:21:05.440328 15508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:21:05.445283 15508 catalog_manager.cc:1347] Generated new cluster ID: 5beeb820173948b9874cbc92e8935e27
I20260810 06:21:05.445339 15508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:21:05.455569 15508 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:21:05.456642 15508 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:21:05.464962 15508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 203da7427b9940159752bdbd7fc68a7f: Generated new TSK 0
I20260810 06:21:05.465603 15508 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:21:05.473148 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:46101
--builtin_ntp_servers=127.31.204.212:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 06:21:05.581741 15524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:21:05.581977 15524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:21:05.582047 15524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:21:05.585918 15524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:21:05.585963 15524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:21:05.586048 15524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:21:05.590567 15524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:46101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:21:05.591466 15524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:21:05.592732 15524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:21:05.601679 15535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:05.601652 15532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:05.601686 15533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:05.601951 15524 server_base.cc:1034] running on GCE node
I20260810 06:21:05.602377 15524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:21:05.602936 15524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:21:05.604103 15524 hybrid_clock.cc:648] HybridClock initialized: now 1786342865604086 us; error 31 us; skew 500 ppm
I20260810 06:21:05.605749 15524 webserver.cc:460] Webserver started at http://127.31.204.193:45187/ using document root <none> and password file <none>
I20260810 06:21:05.606307 15524 fs_manager.cc:359] Metadata directory not provided
I20260810 06:21:05.606376 15524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:21:05.606580 15524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:21:05.608654 15524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "06d4a1c46f674995b412e4d73c3366e3"
format_stamp: "Formatted at 2026-08-10 06:21:05 on dist-test-slave-q9h9"
server_key: "c6e21cde9b8e63c7f8d5f8dba99b6e20"
server_key_iv: "dfdd209460e1b23fec37dc6ed7333e49"
server_key_version: "encryptionkey@0"
I20260810 06:21:05.609233 15524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "06d4a1c46f674995b412e4d73c3366e3"
format_stamp: "Formatted at 2026-08-10 06:21:05 on dist-test-slave-q9h9"
server_key: "c6e21cde9b8e63c7f8d5f8dba99b6e20"
server_key_iv: "dfdd209460e1b23fec37dc6ed7333e49"
server_key_version: "encryptionkey@0"
I20260810 06:21:05.613092 15524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 06:21:05.615536 15544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:05.616640 15524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:21:05.616755 15524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "06d4a1c46f674995b412e4d73c3366e3"
format_stamp: "Formatted at 2026-08-10 06:21:05 on dist-test-slave-q9h9"
server_key: "c6e21cde9b8e63c7f8d5f8dba99b6e20"
server_key_iv: "dfdd209460e1b23fec37dc6ed7333e49"
server_key_version: "encryptionkey@0"
I20260810 06:21:05.616847 15524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:21:05.639945 15524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:21:05.640515 15524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:21:05.640668 15524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:21:05.641178 15524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:21:05.642067 15524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:21:05.642115 15524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:05.642166 15524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:21:05.642190 15524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:05.650297 15524 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:34079
I20260810 06:21:05.650349 15718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:34079 every 8 connection(s)
I20260810 06:21:05.651160 15524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:21:05.653221 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 15524
I20260810 06:21:05.653333 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:21:05.653651 32563 external_mini_cluster.cc:1428] Setting key ecc836f4b1a449edd2ffd2f183b1440a
I20260810 06:21:05.655759 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:46101
--builtin_ntp_servers=127.31.204.212:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 06:21:05.660287 15481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:05.652834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:40433 (local address 127.31.204.254:46101)
0810 06:21:05.652964 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:05.652968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:05.654043 (+ 1075us) server_negotiation.cc:407] Connection header received
0810 06:21:05.655519 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:05.655521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:05.655587 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:05.655665 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:05.656707 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:05.657240 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:05.658455 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:05.658747 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:05.658841 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:05.659783 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:21:05.659787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:21:05.659791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:21:05.659807 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:21:05.659831 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:21:05.659842 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:21:05.659920 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:05.660031 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 06:21:05.660125 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 06:21:05.660944 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:05.653168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46101 (local address 127.31.204.193:40433)
0810 06:21:05.653850 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:05.653899 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:05.655302 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:05.655862 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:05.655871 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:05.655975 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:05.656560 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:05.656577 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:05.657375 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:05.657378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:05.658338 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:05.658348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:05.658868 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:05.658870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:05.658997 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:05.659536 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:21:05.659538 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:05.659574 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:21:05.659632 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:21:05.659922 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:21:05.659927 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:21:05.659930 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 06:21:05.660010 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 06:21:05.660206 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":118,"threads_started":1}
I20260810 06:21:05.662297 15719 heartbeater.cc:347] Connected to a master server at 127.31.204.254:46101
I20260810 06:21:05.662510 15719 heartbeater.cc:464] Registering TS with master...
I20260810 06:21:05.662986 15719 heartbeater.cc:511] Master 127.31.204.254:46101 requested a full tablet report, sending...
I20260810 06:21:05.664386 15417 ts_manager.cc:194] Registered new tserver with Master: 06d4a1c46f674995b412e4d73c3366e3 (127.31.204.193:34079)
I20260810 06:21:05.665489 15417 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.204.193:40433
W20260810 06:21:05.761782 15722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:21:05.761992 15722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:21:05.762063 15722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:21:05.765849 15722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:21:05.765893 15722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:21:05.765962 15722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:21:05.770378 15722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:46101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:21:05.771198 15722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:21:05.772374 15722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:21:05.780934 15731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:05.780956 15737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:05.780990 15732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:05.781180 15722 server_base.cc:1034] running on GCE node
I20260810 06:21:05.781648 15722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:21:05.782223 15722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:21:05.783380 15722 hybrid_clock.cc:648] HybridClock initialized: now 1786342865783367 us; error 33 us; skew 500 ppm
I20260810 06:21:05.784893 15722 webserver.cc:460] Webserver started at http://127.31.204.194:40479/ using document root <none> and password file <none>
I20260810 06:21:05.785362 15722 fs_manager.cc:359] Metadata directory not provided
I20260810 06:21:05.785416 15722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:21:05.785567 15722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:21:05.787421 15722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "037e39ccbba3472d82e4e326bbc102d5"
format_stamp: "Formatted at 2026-08-10 06:21:05 on dist-test-slave-q9h9"
server_key: "3262653cd6287a13417b9e8a761c81b5"
server_key_iv: "a34ddbe030e0ef0b0d4eec72cf6ffa01"
server_key_version: "encryptionkey@0"
I20260810 06:21:05.787874 15722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "037e39ccbba3472d82e4e326bbc102d5"
format_stamp: "Formatted at 2026-08-10 06:21:05 on dist-test-slave-q9h9"
server_key: "3262653cd6287a13417b9e8a761c81b5"
server_key_iv: "a34ddbe030e0ef0b0d4eec72cf6ffa01"
server_key_version: "encryptionkey@0"
I20260810 06:21:05.791080 15722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:21:05.793223 15750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:05.794304 15722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:21:05.794392 15722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "037e39ccbba3472d82e4e326bbc102d5"
format_stamp: "Formatted at 2026-08-10 06:21:05 on dist-test-slave-q9h9"
server_key: "3262653cd6287a13417b9e8a761c81b5"
server_key_iv: "a34ddbe030e0ef0b0d4eec72cf6ffa01"
server_key_version: "encryptionkey@0"
I20260810 06:21:05.794484 15722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:21:05.803800 15722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:21:05.804352 15722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:21:05.804479 15722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:21:05.804920 15722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:21:05.805684 15722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:21:05.805727 15722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:05.805763 15722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:21:05.805783 15722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:05.813655 15722 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:45449
I20260810 06:21:05.813683 15931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:45449 every 8 connection(s)
I20260810 06:21:05.814429 15722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:21:05.822561 15481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:05.815958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:37427 (local address 127.31.204.254:46101)
0810 06:21:05.816086 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:05.816089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:05.816989 (+ 900us) server_negotiation.cc:407] Connection header received
0810 06:21:05.818256 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:05.818259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:05.818318 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:05.818389 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:05.819182 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:05.819688 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:05.820830 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:05.821042 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:05.821137 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:05.822116 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:21:05.822119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:21:05.822122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:21:05.822134 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:21:05.822153 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:21:05.822160 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:21:05.822219 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:05.822354 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 06:21:05.822416 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:21:05.823316 15941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:05.816254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46101 (local address 127.31.204.194:37427)
0810 06:21:05.816847 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:05.816884 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:05.818064 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:05.818546 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:05.818552 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:05.818635 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:05.819047 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:05.819059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:05.819802 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:05.819804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:05.820717 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:05.820727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:05.821185 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:05.821189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:05.821331 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:05.821846 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:21:05.821848 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:05.821883 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:21:05.821939 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:21:05.822228 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:21:05.822237 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:21:05.822243 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 06:21:05.822345 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 06:21:05.822597 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":110,"threads_started":1}
I20260810 06:21:05.823367 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 15722
I20260810 06:21:05.823463 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:21:05.823791 32563 external_mini_cluster.cc:1428] Setting key 18484f16fc0250396b51b4a05c36ab9f
I20260810 06:21:05.824349 15936 heartbeater.cc:347] Connected to a master server at 127.31.204.254:46101
I20260810 06:21:05.824548 15936 heartbeater.cc:464] Registering TS with master...
I20260810 06:21:05.825040 15936 heartbeater.cc:511] Master 127.31.204.254:46101 requested a full tablet report, sending...
I20260810 06:21:05.825824 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:46101
--builtin_ntp_servers=127.31.204.212:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 06:21:05.826079 15417 ts_manager.cc:194] Registered new tserver with Master: 037e39ccbba3472d82e4e326bbc102d5 (127.31.204.194:45449)
I20260810 06:21:05.826723 15417 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.204.194:37427
W20260810 06:21:05.931718 15942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:21:05.931934 15942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:21:05.931969 15942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:21:05.935889 15942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:21:05.935932 15942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:21:05.935989 15942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:21:05.940536 15942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:46101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:21:05.941404 15942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:21:05.942652 15942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:21:05.951277 15957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:05.951278 15952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:05.951310 15953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:05.951419 15942 server_base.cc:1034] running on GCE node
I20260810 06:21:05.951913 15942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:21:05.952472 15942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:21:05.953626 15942 hybrid_clock.cc:648] HybridClock initialized: now 1786342865953603 us; error 34 us; skew 500 ppm
I20260810 06:21:05.955274 15942 webserver.cc:460] Webserver started at http://127.31.204.195:34449/ using document root <none> and password file <none>
I20260810 06:21:05.955775 15942 fs_manager.cc:359] Metadata directory not provided
I20260810 06:21:05.955833 15942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:21:05.956019 15942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:21:05.957883 15942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "083595281700471bb1f982e7d95339b8"
format_stamp: "Formatted at 2026-08-10 06:21:05 on dist-test-slave-q9h9"
server_key: "310b872b215699270e028be472d0431e"
server_key_iv: "c16bb7e728a4ed8ca1e8ccfe29be18c8"
server_key_version: "encryptionkey@0"
I20260810 06:21:05.958437 15942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "083595281700471bb1f982e7d95339b8"
format_stamp: "Formatted at 2026-08-10 06:21:05 on dist-test-slave-q9h9"
server_key: "310b872b215699270e028be472d0431e"
server_key_iv: "c16bb7e728a4ed8ca1e8ccfe29be18c8"
server_key_version: "encryptionkey@0"
I20260810 06:21:05.961834 15942 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:21:05.964028 15967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:05.965181 15942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:21:05.965276 15942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "083595281700471bb1f982e7d95339b8"
format_stamp: "Formatted at 2026-08-10 06:21:05 on dist-test-slave-q9h9"
server_key: "310b872b215699270e028be472d0431e"
server_key_iv: "c16bb7e728a4ed8ca1e8ccfe29be18c8"
server_key_version: "encryptionkey@0"
I20260810 06:21:05.965364 15942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:21:05.983503 15942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:21:05.984050 15942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:21:05.984310 15942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:21:05.984967 15942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:21:05.985865 15942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:21:05.985910 15942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:05.985949 15942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:21:05.985977 15942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:05.993914 15942 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:46177
I20260810 06:21:05.993937 16151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:46177 every 8 connection(s)
I20260810 06:21:05.994707 15942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:21:06.001989 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 15942
I20260810 06:21:06.002122 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:21:06.002384 32563 external_mini_cluster.cc:1428] Setting key 1b21ad010b7cb30d2428a1ce58fa6934
I20260810 06:21:06.003612 15481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:05.996343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:33243 (local address 127.31.204.254:46101)
0810 06:21:05.997440 (+ 1097us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:05.997444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:05.997698 (+ 254us) server_negotiation.cc:407] Connection header received
0810 06:21:05.999156 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:05.999159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:05.999215 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:05.999292 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:06.000096 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:06.000618 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:06.001679 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:06.001971 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:06.002122 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:06.003148 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:21:06.003152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:21:06.003155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:21:06.003167 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:21:06.003190 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:21:06.003201 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:21:06.003261 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:06.003414 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 06:21:06.003474 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1006}
I20260810 06:21:06.004515 16155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:05.996746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46101 (local address 127.31.204.195:33243)
0810 06:21:05.997576 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:05.997611 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:05.998963 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:05.999450 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:05.999456 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:05.999532 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:05.999962 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:05.999973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:06.000733 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:06.000736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:06.001563 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:06.001573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:06.002166 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:06.002172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:06.002333 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:06.002887 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:21:06.002889 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:06.002924 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:21:06.002996 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:21:06.003270 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:21:06.003279 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:21:06.003285 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 06:21:06.003417 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 06:21:06.003659 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":97,"threads_started":1}
I20260810 06:21:06.005417 16152 heartbeater.cc:347] Connected to a master server at 127.31.204.254:46101
I20260810 06:21:06.005594 16152 heartbeater.cc:464] Registering TS with master...
I20260810 06:21:06.006000 16152 heartbeater.cc:511] Master 127.31.204.254:46101 requested a full tablet report, sending...
I20260810 06:21:06.006914 15417 ts_manager.cc:194] Registered new tserver with Master: 083595281700471bb1f982e7d95339b8 (127.31.204.195:46177)
I20260810 06:21:06.007429 15417 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.204.195:33243
I20260810 06:21:06.015847 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 06:21:06.018355 15481 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:21:06.017528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53240 (local address 127.31.204.254:46101)
0810 06:21:06.017642 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:06.017646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:06.017805 (+ 159us) server_negotiation.cc:407] Connection header received
0810 06:21:06.017898 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:06.017900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:06.017936 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:06.017995 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:06.018242 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53240: BlockingRecv error: recv got EOF from 127.0.0.1:53240 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:21:06.018496 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 15524
I20260810 06:21:06.029639 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 15722
I20260810 06:21:06.037748 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 15942
I20260810 06:21:06.045295 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 15363
2026-08-10T06:21:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (826 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T06:21:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:21:06Z Disabled control of system clock
I20260810 06:21:06.078347 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36575
--webserver_interface=127.31.204.254
--webserver_port=0
--builtin_ntp_servers=127.31.204.212:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:36575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 06:21:06.182423 16168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:21:06.182626 16168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:21:06.182658 16168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:21:06.186471 16168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:21:06.186515 16168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:21:06.186529 16168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:21:06.186544 16168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:21:06.186563 16168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:21:06.191035 16168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:36575
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:21:06.191836 16168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:21:06.193027 16168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:21:06.200685 16179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:06.200598 16182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:06.200587 16186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:06.200816 16168 server_base.cc:1034] running on GCE node
I20260810 06:21:06.201385 16168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:21:06.202450 16168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:21:06.203588 16168 hybrid_clock.cc:648] HybridClock initialized: now 1786342866203563 us; error 45 us; skew 500 ppm
I20260810 06:21:06.205228 16168 webserver.cc:460] Webserver started at http://127.31.204.254:35639/ using document root <none> and password file <none>
I20260810 06:21:06.205721 16168 fs_manager.cc:359] Metadata directory not provided
I20260810 06:21:06.205781 16168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:21:06.205966 16168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:21:06.207819 16168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "58a91c036d794a9583695f017bb2e900"
format_stamp: "Formatted at 2026-08-10 06:21:06 on dist-test-slave-q9h9"
server_key: "7b391b2e9a84c30553549c12d03f9a94"
server_key_iv: "5879434f7a6d8e71ad3f13238515b0cc"
server_key_version: "encryptionkey@0"
I20260810 06:21:06.208340 16168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "58a91c036d794a9583695f017bb2e900"
format_stamp: "Formatted at 2026-08-10 06:21:06 on dist-test-slave-q9h9"
server_key: "7b391b2e9a84c30553549c12d03f9a94"
server_key_iv: "5879434f7a6d8e71ad3f13238515b0cc"
server_key_version: "encryptionkey@0"
I20260810 06:21:06.211594 16168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:21:06.213662 16198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:06.214759 16168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:21:06.214886 16168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "58a91c036d794a9583695f017bb2e900"
format_stamp: "Formatted at 2026-08-10 06:21:06 on dist-test-slave-q9h9"
server_key: "7b391b2e9a84c30553549c12d03f9a94"
server_key_iv: "5879434f7a6d8e71ad3f13238515b0cc"
server_key_version: "encryptionkey@0"
I20260810 06:21:06.214973 16168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:21:06.219108 16168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:21:06.219635 16168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:21:06.219781 16168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:21:06.225960 16168 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:36575
I20260810 06:21:06.225983 16285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:36575 every 8 connection(s)
I20260810 06:21:06.226783 16168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:21:06.229676 16289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:21:06.233140 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 16168
I20260810 06:21:06.233253 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:21:06.233505 32563 external_mini_cluster.cc:1428] Setting key 51133104b0aee92f797eb638fa15b0be
I20260810 06:21:06.234944 16289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900: Bootstrap starting.
I20260810 06:21:06.237498 16289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900: Neither blocks nor log segments found. Creating new log.
I20260810 06:21:06.238372 16289 log.cc:824] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900: Log is configured to *not* fsync() on all Append() calls
I20260810 06:21:06.239076 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:06.234844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34422 (local address 127.31.204.254:36575)
0810 06:21:06.235459 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:06.235480 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:06.235518 (+ 38us) server_negotiation.cc:407] Connection header received
0810 06:21:06.236321 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:06.236342 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:06.236754 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:06.237080 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:06.237765 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:21:06.237771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:21:06.237786 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:21:06.237796 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:21:06.237863 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:21:06.237885 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:21:06.237990 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:06.238086 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 06:21:06.238315 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":95,"threads_started":1}
I20260810 06:21:06.240154 16289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900: No bootstrap required, opened a new log
I20260810 06:21:06.243690 16289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a91c036d794a9583695f017bb2e900" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36575 } }
I20260810 06:21:06.243892 16289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:21:06.243960 16289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a91c036d794a9583695f017bb2e900, State: Initialized, Role: FOLLOWER
I20260810 06:21:06.244361 16289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a91c036d794a9583695f017bb2e900" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36575 } }
I20260810 06:21:06.244489 16289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:21:06.244549 16289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:21:06.244642 16289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:21:06.245460 16289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a91c036d794a9583695f017bb2e900" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36575 } }
I20260810 06:21:06.245744 16289 leader_election.cc:302] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58a91c036d794a9583695f017bb2e900; no voters:
I20260810 06:21:06.245993 16289 leader_election.cc:288] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:21:06.246112 16298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:21:06.246309 16298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900 [term 1 LEADER]: Becoming Leader. State: Replica: 58a91c036d794a9583695f017bb2e900, State: Running, Role: LEADER
I20260810 06:21:06.246584 16298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a91c036d794a9583695f017bb2e900" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36575 } }
I20260810 06:21:06.247174 16289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:21:06.247614 16301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58a91c036d794a9583695f017bb2e900. Latest consensus state: current_term: 1 leader_uuid: "58a91c036d794a9583695f017bb2e900" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a91c036d794a9583695f017bb2e900" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36575 } } }
I20260810 06:21:06.247740 16301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900 [sys.catalog]: This master's current role is: LEADER
I20260810 06:21:06.248008 16302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58a91c036d794a9583695f017bb2e900" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a91c036d794a9583695f017bb2e900" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36575 } } }
I20260810 06:21:06.248099 16302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900 [sys.catalog]: This master's current role is: LEADER
I20260810 06:21:06.248346 16305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:21:06.250888 16305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:21:06.255460 16305 catalog_manager.cc:1347] Generated new cluster ID: 7bd047b3197948eb96849d92e9dee4c8
I20260810 06:21:06.255519 16305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:21:06.278530 16305 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:21:06.278842 16305 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:21:06.288429 16305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58a91c036d794a9583695f017bb2e900: Generated new TSK 0
I20260810 06:21:06.289122 16305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:21:06.294682 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36575
--builtin_ntp_servers=127.31.204.212:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 06:21:06.402834 16327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:21:06.403061 16327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:21:06.403100 16327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:21:06.406939 16327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:21:06.406981 16327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:21:06.407032 16327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:21:06.411535 16327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:21:06.412433 16327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:21:06.413689 16327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:21:06.422725 16342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:06.422847 16336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:06.422725 16334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:06.422991 16327 server_base.cc:1034] running on GCE node
I20260810 06:21:06.423477 16327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:21:06.424057 16327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:21:06.425215 16327 hybrid_clock.cc:648] HybridClock initialized: now 1786342866425195 us; error 30 us; skew 500 ppm
I20260810 06:21:06.426910 16327 webserver.cc:460] Webserver started at http://127.31.204.193:35049/ using document root <none> and password file <none>
I20260810 06:21:06.427433 16327 fs_manager.cc:359] Metadata directory not provided
I20260810 06:21:06.427493 16327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:21:06.427693 16327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:21:06.429612 16327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "82f8aad5072b40268531bf82530b7fc4"
format_stamp: "Formatted at 2026-08-10 06:21:06 on dist-test-slave-q9h9"
server_key: "f511cfa1ee45093ba7bd0d505a7d0a57"
server_key_iv: "1736dcffdf86b2e02a3420f8adf468af"
server_key_version: "encryptionkey@0"
I20260810 06:21:06.430166 16327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "82f8aad5072b40268531bf82530b7fc4"
format_stamp: "Formatted at 2026-08-10 06:21:06 on dist-test-slave-q9h9"
server_key: "f511cfa1ee45093ba7bd0d505a7d0a57"
server_key_iv: "1736dcffdf86b2e02a3420f8adf468af"
server_key_version: "encryptionkey@0"
I20260810 06:21:06.433566 16327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:21:06.435935 16350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:06.437044 16327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 06:21:06.437165 16327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "82f8aad5072b40268531bf82530b7fc4"
format_stamp: "Formatted at 2026-08-10 06:21:06 on dist-test-slave-q9h9"
server_key: "f511cfa1ee45093ba7bd0d505a7d0a57"
server_key_iv: "1736dcffdf86b2e02a3420f8adf468af"
server_key_version: "encryptionkey@0"
I20260810 06:21:06.437255 16327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:21:06.441499 16327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:21:06.442078 16327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:21:06.442235 16327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:21:06.442751 16327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:21:06.443600 16327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:21:06.443646 16327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:06.443694 16327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:21:06.443723 16327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:06.451836 16327 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:42743
I20260810 06:21:06.451861 16552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:42743 every 8 connection(s)
I20260810 06:21:06.452631 16327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:21:06.457931 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:06.454232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:44063 (local address 127.31.204.254:36575)
0810 06:21:06.454372 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:06.454376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:06.455385 (+ 1009us) server_negotiation.cc:407] Connection header received
0810 06:21:06.456580 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:06.456582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:06.456619 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:06.456690 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:06.457597 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:21:06.457600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:21:06.457602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:21:06.457613 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:21:06.457631 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:21:06.457639 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:21:06.457706 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:06.457793 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 06:21:06.457826 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:21:06.458741 16555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:06.454601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36575 (local address 127.31.204.193:44063)
0810 06:21:06.455248 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:06.455287 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:06.456413 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:06.456829 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:06.456835 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:06.456916 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:06.457371 (+ 455us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:21:06.457372 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:06.457409 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:21:06.457470 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:21:06.457697 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:21:06.457702 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:21:06.457705 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 06:21:06.457819 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 06:21:06.458002 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":132,"threads_started":1}
I20260810 06:21:06.459820 16553 heartbeater.cc:347] Connected to a master server at 127.31.204.254:36575
I20260810 06:21:06.460018 16553 heartbeater.cc:464] Registering TS with master...
I20260810 06:21:06.460384 16553 heartbeater.cc:511] Master 127.31.204.254:36575 requested a full tablet report, sending...
I20260810 06:21:06.460954 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 16327
I20260810 06:21:06.461084 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:21:06.461390 32563 external_mini_cluster.cc:1428] Setting key df3be58bc46f23118d97277a7057207d
I20260810 06:21:06.461858 16223 ts_manager.cc:194] Registered new tserver with Master: 82f8aad5072b40268531bf82530b7fc4 (127.31.204.193:42743)
I20260810 06:21:06.463433 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36575
--builtin_ntp_servers=127.31.204.212:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 06:21:06.570415 16556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:21:06.570623 16556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:21:06.570657 16556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:21:06.574493 16556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:21:06.574533 16556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:21:06.574587 16556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:21:06.579048 16556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:21:06.579928 16556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:21:06.581153 16556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:21:06.589915 16565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:06.589957 16563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:06.590004 16556 server_base.cc:1034] running on GCE node
W20260810 06:21:06.589954 16567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:06.590605 16556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:21:06.591207 16556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:21:06.592368 16556 hybrid_clock.cc:648] HybridClock initialized: now 1786342866592350 us; error 28 us; skew 500 ppm
I20260810 06:21:06.594055 16556 webserver.cc:460] Webserver started at http://127.31.204.194:38681/ using document root <none> and password file <none>
I20260810 06:21:06.594584 16556 fs_manager.cc:359] Metadata directory not provided
I20260810 06:21:06.594645 16556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:21:06.594851 16556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:21:06.596786 16556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "2ca996b70aa344cabb6089072ff0b7ba"
format_stamp: "Formatted at 2026-08-10 06:21:06 on dist-test-slave-q9h9"
server_key: "e3353ae4024bb335acb412d164a36036"
server_key_iv: "c306c7f73facd131b943bf69143e9cd1"
server_key_version: "encryptionkey@0"
I20260810 06:21:06.597307 16556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "2ca996b70aa344cabb6089072ff0b7ba"
format_stamp: "Formatted at 2026-08-10 06:21:06 on dist-test-slave-q9h9"
server_key: "e3353ae4024bb335acb412d164a36036"
server_key_iv: "c306c7f73facd131b943bf69143e9cd1"
server_key_version: "encryptionkey@0"
I20260810 06:21:06.601042 16556 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 06:21:06.603443 16575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:06.604538 16556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:21:06.604653 16556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "2ca996b70aa344cabb6089072ff0b7ba"
format_stamp: "Formatted at 2026-08-10 06:21:06 on dist-test-slave-q9h9"
server_key: "e3353ae4024bb335acb412d164a36036"
server_key_iv: "c306c7f73facd131b943bf69143e9cd1"
server_key_version: "encryptionkey@0"
I20260810 06:21:06.604756 16556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:21:06.609476 16556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:21:06.610261 16556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:21:06.610445 16556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:21:06.611016 16556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:21:06.612005 16556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:21:06.612054 16556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:06.612106 16556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:21:06.612136 16556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:06.620667 16556 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:46467
I20260810 06:21:06.620718 16772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:46467 every 8 connection(s)
I20260810 06:21:06.621516 16556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:21:06.627228 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:06.623161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:53713 (local address 127.31.204.254:36575)
0810 06:21:06.623286 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:06.623289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:06.624289 (+ 1000us) server_negotiation.cc:407] Connection header received
0810 06:21:06.625588 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:06.625592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:06.625639 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:06.625714 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:06.626767 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:21:06.626771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:21:06.626774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:21:06.626787 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:21:06.626807 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:21:06.626815 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:21:06.626893 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:06.627019 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 06:21:06.627065 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:21:06.627976 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:06.623521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36575 (local address 127.31.204.194:53713)
0810 06:21:06.624148 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:06.624187 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:06.625364 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:06.625881 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:06.625888 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:06.625971 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:06.626510 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:21:06.626511 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:06.626550 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:21:06.626614 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:21:06.626885 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:21:06.626894 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:21:06.626900 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 06:21:06.627051 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 06:21:06.627289 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":116,"threads_started":1}
I20260810 06:21:06.628938 16774 heartbeater.cc:347] Connected to a master server at 127.31.204.254:36575
I20260810 06:21:06.629114 16774 heartbeater.cc:464] Registering TS with master...
I20260810 06:21:06.629457 16774 heartbeater.cc:511] Master 127.31.204.254:36575 requested a full tablet report, sending...
I20260810 06:21:06.629884 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 16556
I20260810 06:21:06.630028 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:21:06.630342 32563 external_mini_cluster.cc:1428] Setting key c91f10ce2861991f869e38fb4e894a1c
I20260810 06:21:06.630508 16223 ts_manager.cc:194] Registered new tserver with Master: 2ca996b70aa344cabb6089072ff0b7ba (127.31.204.194:46467)
I20260810 06:21:06.632148 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36575
--builtin_ntp_servers=127.31.204.212:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 06:21:06.737843 16783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:21:06.738086 16783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:21:06.738129 16783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:21:06.742033 16783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:21:06.742077 16783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:21:06.742139 16783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:21:06.746603 16783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:21:06.747521 16783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:21:06.748816 16783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:21:06.757638 16791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:06.757716 16793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:06.757828 16783 server_base.cc:1034] running on GCE node
W20260810 06:21:06.757683 16795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:06.758302 16783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:21:06.758864 16783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:21:06.760011 16783 hybrid_clock.cc:648] HybridClock initialized: now 1786342866759999 us; error 24 us; skew 500 ppm
I20260810 06:21:06.761593 16783 webserver.cc:460] Webserver started at http://127.31.204.195:37335/ using document root <none> and password file <none>
I20260810 06:21:06.762177 16783 fs_manager.cc:359] Metadata directory not provided
I20260810 06:21:06.762250 16783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:21:06.762446 16783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:21:06.764391 16783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "3675c66fd3184f76ac9ec2550e20cf8a"
format_stamp: "Formatted at 2026-08-10 06:21:06 on dist-test-slave-q9h9"
server_key: "b52cb3f42d0c6e328d6800a7f234efc2"
server_key_iv: "35341c1e37fa3c9495841960d2980074"
server_key_version: "encryptionkey@0"
I20260810 06:21:06.764927 16783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "3675c66fd3184f76ac9ec2550e20cf8a"
format_stamp: "Formatted at 2026-08-10 06:21:06 on dist-test-slave-q9h9"
server_key: "b52cb3f42d0c6e328d6800a7f234efc2"
server_key_iv: "35341c1e37fa3c9495841960d2980074"
server_key_version: "encryptionkey@0"
I20260810 06:21:06.768671 16783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 06:21:06.771320 16801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:06.772593 16783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:21:06.772698 16783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "3675c66fd3184f76ac9ec2550e20cf8a"
format_stamp: "Formatted at 2026-08-10 06:21:06 on dist-test-slave-q9h9"
server_key: "b52cb3f42d0c6e328d6800a7f234efc2"
server_key_iv: "35341c1e37fa3c9495841960d2980074"
server_key_version: "encryptionkey@0"
I20260810 06:21:06.772810 16783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:21:06.777458 16783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:21:06.778128 16783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:21:06.778294 16783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:21:06.778879 16783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:21:06.779812 16783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:21:06.779858 16783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:06.779932 16783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:21:06.779958 16783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:06.788563 16783 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:37375
I20260810 06:21:06.788606 16978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:37375 every 8 connection(s)
I20260810 06:21:06.789534 16783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:21:06.795708 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:06.791400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:54323 (local address 127.31.204.254:36575)
0810 06:21:06.791540 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:06.791544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:06.792673 (+ 1129us) server_negotiation.cc:407] Connection header received
0810 06:21:06.794200 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:06.794203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:06.794244 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:06.794310 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:06.795330 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:21:06.795333 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:21:06.795335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:21:06.795346 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:21:06.795366 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:21:06.795374 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:21:06.795448 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:06.795549 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 06:21:06.795586 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 06:21:06.796667 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:06.791794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36575 (local address 127.31.204.195:54323)
0810 06:21:06.792526 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:06.792571 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:06.793921 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:06.794445 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:06.794451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:06.794553 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:06.795075 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:21:06.795077 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:06.795117 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:21:06.795190 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:21:06.795442 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:21:06.795447 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:21:06.795451 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 06:21:06.795600 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 06:21:06.795846 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":122,"threads_started":1}
I20260810 06:21:06.797855 16980 heartbeater.cc:347] Connected to a master server at 127.31.204.254:36575
I20260810 06:21:06.798094 16980 heartbeater.cc:464] Registering TS with master...
I20260810 06:21:06.798138 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 16783
I20260810 06:21:06.798285 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:21:06.798554 16980 heartbeater.cc:511] Master 127.31.204.254:36575 requested a full tablet report, sending...
I20260810 06:21:06.798610 32563 external_mini_cluster.cc:1428] Setting key 9f0699de07264418a7422a8dd81ec5e8
I20260810 06:21:06.799515 16223 ts_manager.cc:194] Registered new tserver with Master: 3675c66fd3184f76ac9ec2550e20cf8a (127.31.204.195:37375)
I20260810 06:21:06.800614 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 06:21:06.803568 16295 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:21:06.802724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34438 (local address 127.31.204.254:36575)
0810 06:21:06.802848 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:06.802851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:06.803043 (+ 192us) server_negotiation.cc:407] Connection header received
0810 06:21:06.803148 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:06.803150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:06.803182 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:06.803247 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:06.803450 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34438: BlockingRecv error: recv got EOF from 127.0.0.1:34438 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 06:21:06.803939 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 16327
I20260810 06:21:06.823808 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 16556
I20260810 06:21:06.833180 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 16783
I20260810 06:21:06.845853 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 16168
2026-08-10T06:21:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (798 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:21:06 dist-test-slave-q9h9 krb5kdc[17000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:21:06 dist-test-slave-q9h9 krb5kdc[17000](info): set up 2 sockets
Aug 10 06:21:06 dist-test-slave-q9h9 krb5kdc[17000](info): commencing operation
krb5kdc: starting...
W20260810 06:21:08.003118 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:21:08 dist-test-slave-q9h9 krb5kdc[17000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:21:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:21:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:21:08.110771 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:40559
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:42647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:40559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:21:08.215065 17019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:21:08.215365 17019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:21:08.215430 17019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:21:08.219334 17019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:21:08.219381 17019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:21:08.219399 17019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:21:08.219417 17019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:21:08.219445 17019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:21:08.224788 17019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:42647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:40559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:40559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:21:08.225710 17019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:21:08.226683 17019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:21:08.234326 17034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:08.234386 17032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:08.234386 17029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:08.234550 17019 server_base.cc:1034] running on GCE node
I20260810 06:21:08.235076 17019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:21:08.236017 17019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:21:08.237155 17019 hybrid_clock.cc:648] HybridClock initialized: now 1786342868237136 us; error 35 us; skew 500 ppm
Aug 10 06:21:08 dist-test-slave-q9h9 krb5kdc[17000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342868, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:21:08.244055 17019 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:21:08.244870 17019 webserver.cc:460] Webserver started at http://127.31.204.254:37663/ using document root <none> and password file <none>
I20260810 06:21:08.245362 17019 fs_manager.cc:359] Metadata directory not provided
I20260810 06:21:08.245426 17019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:21:08.245610 17019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:21:08.247511 17019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "dc060fa01a2c45468d883f7fc2336ea5"
format_stamp: "Formatted at 2026-08-10 06:21:08 on dist-test-slave-q9h9"
server_key: "cace913e1c090ca7dd38d38c764adf40"
server_key_iv: "1faf01ce0ed781e95d6756f4deeaa525"
server_key_version: "encryptionkey@0"
I20260810 06:21:08.248004 17019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "dc060fa01a2c45468d883f7fc2336ea5"
format_stamp: "Formatted at 2026-08-10 06:21:08 on dist-test-slave-q9h9"
server_key: "cace913e1c090ca7dd38d38c764adf40"
server_key_iv: "1faf01ce0ed781e95d6756f4deeaa525"
server_key_version: "encryptionkey@0"
I20260810 06:21:08.251348 17019 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:21:08.253461 17050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:08.254573 17019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:21:08.254679 17019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "dc060fa01a2c45468d883f7fc2336ea5"
format_stamp: "Formatted at 2026-08-10 06:21:08 on dist-test-slave-q9h9"
server_key: "cace913e1c090ca7dd38d38c764adf40"
server_key_iv: "1faf01ce0ed781e95d6756f4deeaa525"
server_key_version: "encryptionkey@0"
I20260810 06:21:08.254762 17019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:21:08.279893 17019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:21:08.282305 17019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:21:08.282457 17019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:21:08.288851 17019 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:40559
I20260810 06:21:08.289115 17148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:40559 every 8 connection(s)
I20260810 06:21:08.289691 17019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:21:08.292351 17150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:21:08.295621 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 17019
I20260810 06:21:08.295727 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:21:08.295975 32563 external_mini_cluster.cc:1428] Setting key e0e4bb143623268df712f9a65c60f56a
I20260810 06:21:08.297899 17150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5: Bootstrap starting.
I20260810 06:21:08.300522 17150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5: Neither blocks nor log segments found. Creating new log.
I20260810 06:21:08.301364 17150 log.cc:824] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5: Log is configured to *not* fsync() on all Append() calls
I20260810 06:21:08.303478 17150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5: No bootstrap required, opened a new log
Aug 10 06:21:08 dist-test-slave-q9h9 krb5kdc[17000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:21:08.307300 17150 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc060fa01a2c45468d883f7fc2336ea5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 40559 } }
I20260810 06:21:08.307564 17150 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:21:08.307641 17150 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc060fa01a2c45468d883f7fc2336ea5, State: Initialized, Role: FOLLOWER
I20260810 06:21:08.308059 17150 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc060fa01a2c45468d883f7fc2336ea5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 40559 } }
I20260810 06:21:08.308219 17150 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:21:08.308287 17150 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:21:08.308378 17150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:21:08.309348 17150 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc060fa01a2c45468d883f7fc2336ea5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 40559 } }
I20260810 06:21:08.309659 17150 leader_election.cc:302] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc060fa01a2c45468d883f7fc2336ea5; no voters:
I20260810 06:21:08.309927 17150 leader_election.cc:288] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:21:08.310047 17156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:21:08.310086 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:08.297220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38548 (local address 127.31.204.254:40559)
0810 06:21:08.297611 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:08.297629 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:08.297669 (+ 40us) server_negotiation.cc:407] Connection header received
0810 06:21:08.298493 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:08.298515 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:08.298897 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:08.299267 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:08.299875 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.301340 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.302360 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.303002 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.303161 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:08.305677 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:21:08.305682 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:21:08.305693 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:21:08.305718 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:21:08.307245 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:08.307738 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:08.307744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:08.307750 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:08.307869 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:08.308165 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:08.308168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:08.308169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:08.308601 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:21:08.308734 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:08.309022 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 06:21:08.309249 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":90,"threads_started":1}
I20260810 06:21:08.310227 17156 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5 [term 1 LEADER]: Becoming Leader. State: Replica: dc060fa01a2c45468d883f7fc2336ea5, State: Running, Role: LEADER
I20260810 06:21:08.310499 17156 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc060fa01a2c45468d883f7fc2336ea5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 40559 } }
I20260810 06:21:08.311162 17150 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:21:08.311659 17161 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc060fa01a2c45468d883f7fc2336ea5. Latest consensus state: current_term: 1 leader_uuid: "dc060fa01a2c45468d883f7fc2336ea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc060fa01a2c45468d883f7fc2336ea5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 40559 } } }
I20260810 06:21:08.311762 17161 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5 [sys.catalog]: This master's current role is: LEADER
I20260810 06:21:08.312085 17158 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc060fa01a2c45468d883f7fc2336ea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc060fa01a2c45468d883f7fc2336ea5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 40559 } } }
I20260810 06:21:08.312191 17158 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5 [sys.catalog]: This master's current role is: LEADER
I20260810 06:21:08.312737 17164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:21:08.315685 17164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:21:08.321107 17164 catalog_manager.cc:1347] Generated new cluster ID: a1049945a18d4eda85dff2b05c83c30e
I20260810 06:21:08.321167 17164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:21:08.335429 17164 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:21:08.336210 17164 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:21:08.346622 17164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5: Generated new TSK 0
I20260810 06:21:08.347149 17164 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
I20260810 06:21:08.373677 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:40559
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:42647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:21:08.479758 17192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:21:08.479979 17192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:21:08.480019 17192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:21:08.483933 17192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:21:08.483986 17192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:21:08.484040 17192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:21:08.488965 17192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:42647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:40559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:21:08.489869 17192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:21:08.490842 17192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:21:08.499763 17199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:08.499797 17192 server_base.cc:1034] running on GCE node
W20260810 06:21:08.499771 17205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:08.499748 17202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:08.500351 17192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:21:08.500907 17192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:21:08.502070 17192 hybrid_clock.cc:648] HybridClock initialized: now 1786342868502055 us; error 26 us; skew 500 ppm
Aug 10 06:21:08 dist-test-slave-q9h9 krb5kdc[17000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342868, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:21:08.504561 17192 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:21:08.505344 17192 webserver.cc:460] Webserver started at http://127.31.204.193:43885/ using document root <none> and password file <none>
I20260810 06:21:08.505854 17192 fs_manager.cc:359] Metadata directory not provided
I20260810 06:21:08.505918 17192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:21:08.506150 17192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:21:08.508033 17192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "77c7a4804d7346a6b12c412324498deb"
format_stamp: "Formatted at 2026-08-10 06:21:08 on dist-test-slave-q9h9"
server_key: "a0124b484d4c56b40d48946f118e188c"
server_key_iv: "c7bb8555e9a947e9b3d552d5ee6517f2"
server_key_version: "encryptionkey@0"
I20260810 06:21:08.508539 17192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "77c7a4804d7346a6b12c412324498deb"
format_stamp: "Formatted at 2026-08-10 06:21:08 on dist-test-slave-q9h9"
server_key: "a0124b484d4c56b40d48946f118e188c"
server_key_iv: "c7bb8555e9a947e9b3d552d5ee6517f2"
server_key_version: "encryptionkey@0"
I20260810 06:21:08.512228 17192 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:21:08.514474 17219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:08.515563 17192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:21:08.515666 17192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "77c7a4804d7346a6b12c412324498deb"
format_stamp: "Formatted at 2026-08-10 06:21:08 on dist-test-slave-q9h9"
server_key: "a0124b484d4c56b40d48946f118e188c"
server_key_iv: "c7bb8555e9a947e9b3d552d5ee6517f2"
server_key_version: "encryptionkey@0"
I20260810 06:21:08.515756 17192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:21:08.527216 17192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:21:08.529601 17192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:21:08.529763 17192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:21:08.530318 17192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:21:08.531057 17192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:21:08.531101 17192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:08.531142 17192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:21:08.531173 17192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:08.539268 17192 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:41301
I20260810 06:21:08.539297 17404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:41301 every 8 connection(s)
I20260810 06:21:08.540056 17192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
Aug 10 06:21:08 dist-test-slave-q9h9 krb5kdc[17000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342868, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:21:08.549082 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 17192
I20260810 06:21:08.549189 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:21:08.549463 32563 external_mini_cluster.cc:1428] Setting key 8a38616267667c9e2762be453ba432a6
I20260810 06:21:08.552542 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:08.541626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:37215 (local address 127.31.204.254:40559)
0810 06:21:08.541736 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:08.541740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:08.542538 (+ 798us) server_negotiation.cc:407] Connection header received
0810 06:21:08.544000 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:08.544004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:08.544063 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:08.544138 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:08.545233 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.545754 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.546870 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.547087 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.547177 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:08.549385 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:21:08.549388 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:21:08.549390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:21:08.549413 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:21:08.550724 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:08.551175 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:08.551179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:08.551181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:08.551255 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:08.551554 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:08.551558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:08.551559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:08.551730 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:21:08.551813 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:08.552270 (+ 457us) server_negotiation.cc:299] Negotiation successful
0810 06:21:08.552388 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 06:21:08.553303 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:08.541926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:40559 (local address 127.31.204.193:37215)
0810 06:21:08.542400 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:08.542440 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:08.543787 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:08.544311 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:08.544317 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:08.544649 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:08.545087 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:08.545098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.545872 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.545874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:08.546749 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:08.546758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.547228 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.547232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:08.547379 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:08.547839 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:08.547855 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:08.549235 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:21:08.550832 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:21:08.550837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:21:08.550851 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:21:08.551084 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:21:08.551335 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:21:08.551338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:21:08.551340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:21:08.551468 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:21:08.551829 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:21:08.551835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:21:08.552049 (+ 214us) client_negotiation.cc:769] Sending connection context
0810 06:21:08.552220 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 06:21:08.552508 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":116,"threads_started":1}
I20260810 06:21:08.554550 17407 heartbeater.cc:347] Connected to a master server at 127.31.204.254:40559
I20260810 06:21:08.554749 17407 heartbeater.cc:464] Registering TS with master...
I20260810 06:21:08.555202 17407 heartbeater.cc:511] Master 127.31.204.254:40559 requested a full tablet report, sending...
I20260810 06:21:08.556736 17090 ts_manager.cc:194] Registered new tserver with Master: 77c7a4804d7346a6b12c412324498deb (127.31.204.193:41301)
I20260810 06:21:08.558310 17090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:37215
WARNING: no policy specified for kudu/127.31.204.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
I20260810 06:21:08.573791 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:40559
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:42647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:21:08.681190 17415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:21:08.681411 17415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:21:08.681452 17415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:21:08.685303 17415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:21:08.685353 17415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:21:08.685405 17415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:21:08.690330 17415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:42647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:40559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:21:08.691188 17415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:21:08.692123 17415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:21:08.700927 17422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:08.700960 17427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:08.701043 17415 server_base.cc:1034] running on GCE node
W20260810 06:21:08.700968 17421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:08.701555 17415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:21:08.702140 17415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:21:08.703289 17415 hybrid_clock.cc:648] HybridClock initialized: now 1786342868703272 us; error 26 us; skew 500 ppm
Aug 10 06:21:08 dist-test-slave-q9h9 krb5kdc[17000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342868, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:21:08.705735 17415 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:21:08.706530 17415 webserver.cc:460] Webserver started at http://127.31.204.194:42455/ using document root <none> and password file <none>
I20260810 06:21:08.707019 17415 fs_manager.cc:359] Metadata directory not provided
I20260810 06:21:08.707079 17415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:21:08.707267 17415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:21:08.709162 17415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "6670ad5d996949269a878c05e77e0f2f"
format_stamp: "Formatted at 2026-08-10 06:21:08 on dist-test-slave-q9h9"
server_key: "709f884920408329c6fd568fd4de937b"
server_key_iv: "d9d5b14869a8ca321b03e939ae4f740c"
server_key_version: "encryptionkey@0"
I20260810 06:21:08.709661 17415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "6670ad5d996949269a878c05e77e0f2f"
format_stamp: "Formatted at 2026-08-10 06:21:08 on dist-test-slave-q9h9"
server_key: "709f884920408329c6fd568fd4de937b"
server_key_iv: "d9d5b14869a8ca321b03e939ae4f740c"
server_key_version: "encryptionkey@0"
I20260810 06:21:08.713178 17415 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:21:08.715382 17436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:08.716450 17415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:21:08.716545 17415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "6670ad5d996949269a878c05e77e0f2f"
format_stamp: "Formatted at 2026-08-10 06:21:08 on dist-test-slave-q9h9"
server_key: "709f884920408329c6fd568fd4de937b"
server_key_iv: "d9d5b14869a8ca321b03e939ae4f740c"
server_key_version: "encryptionkey@0"
I20260810 06:21:08.716634 17415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:21:08.733741 17415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:21:08.736186 17415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:21:08.736341 17415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:21:08.736857 17415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:21:08.737586 17415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:21:08.737629 17415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:08.737669 17415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:21:08.737707 17415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:08.745702 17415 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:40539
I20260810 06:21:08.745721 17637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:40539 every 8 connection(s)
I20260810 06:21:08.746515 17415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:21:08.748914 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 17415
I20260810 06:21:08.749040 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:21:08.749372 32563 external_mini_cluster.cc:1428] Setting key 5ab5a2630a6aa903ecd77ca5fef4b951
Aug 10 06:21:08 dist-test-slave-q9h9 krb5kdc[17000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342868, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.204.195@KRBTEST.COM; defaulting to no policy
I20260810 06:21:08.760296 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:08.748093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:47829 (local address 127.31.204.254:40559)
0810 06:21:08.748229 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:08.748234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:08.749083 (+ 849us) server_negotiation.cc:407] Connection header received
0810 06:21:08.750605 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:08.750609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:08.750654 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:08.750714 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:08.751788 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.752467 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.753971 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.754223 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.754302 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:08.756865 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:21:08.756868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:21:08.756869 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:21:08.756893 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:21:08.758304 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:08.758854 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:08.758856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:08.758857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:08.758935 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:08.759300 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:08.759302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:08.759303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:08.759457 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:21:08.759527 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:08.760068 (+ 541us) server_negotiation.cc:299] Negotiation successful
0810 06:21:08.760173 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 06:21:08.761102 17640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:08.748455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:40559 (local address 127.31.204.194:47829)
0810 06:21:08.748924 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:08.748980 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:08.750425 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:08.750876 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:08.750882 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:08.751206 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:08.751631 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:08.751643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.752564 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.752566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:08.753862 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:08.753873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.754366 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.754370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:08.754547 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:08.755080 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:08.755099 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:08.756705 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:21:08.758419 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:21:08.758425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:21:08.758439 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:21:08.758761 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:21:08.759016 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:21:08.759019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:21:08.759021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:21:08.759193 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:21:08.759557 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:21:08.759564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:21:08.759841 (+ 277us) client_negotiation.cc:769] Sending connection context
0810 06:21:08.760030 (+ 189us) client_negotiation.cc:240] Negotiation successful
0810 06:21:08.760351 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":102,"threads_started":1}
I20260810 06:21:08.762066 17638 heartbeater.cc:347] Connected to a master server at 127.31.204.254:40559
I20260810 06:21:08.762270 17638 heartbeater.cc:464] Registering TS with master...
I20260810 06:21:08.762691 17638 heartbeater.cc:511] Master 127.31.204.254:40559 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.195@KRBTEST.COM" created.
I20260810 06:21:08.764818 17090 ts_manager.cc:194] Registered new tserver with Master: 6670ad5d996949269a878c05e77e0f2f (127.31.204.194:40539)
I20260810 06:21:08.765354 17090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:47829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
I20260810 06:21:08.775102 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:40559
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:42647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:21:08.880057 17644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:21:08.880272 17644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:21:08.880324 17644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:21:08.884153 17644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:21:08.884202 17644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:21:08.884280 17644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:21:08.889184 17644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:42647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:40559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:21:08.890090 17644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:21:08.891232 17644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:21:08.900416 17655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:08.900429 17652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:08.900477 17659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:08.900566 17644 server_base.cc:1034] running on GCE node
I20260810 06:21:08.900936 17644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:21:08.901479 17644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:21:08.902629 17644 hybrid_clock.cc:648] HybridClock initialized: now 1786342868902610 us; error 31 us; skew 500 ppm
Aug 10 06:21:08 dist-test-slave-q9h9 krb5kdc[17000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342868, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:21:08.905030 17644 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:21:08.905763 17644 webserver.cc:460] Webserver started at http://127.31.204.195:35669/ using document root <none> and password file <none>
I20260810 06:21:08.906256 17644 fs_manager.cc:359] Metadata directory not provided
I20260810 06:21:08.906324 17644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:21:08.906481 17644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:21:08.908274 17644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "202121fcadfb44d99070dc9094407551"
format_stamp: "Formatted at 2026-08-10 06:21:08 on dist-test-slave-q9h9"
server_key: "cb8a09b5354eb6fdb2464449be5bbde2"
server_key_iv: "82e33568d082d4e068764ae0cc16cb1c"
server_key_version: "encryptionkey@0"
I20260810 06:21:08.908738 17644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "202121fcadfb44d99070dc9094407551"
format_stamp: "Formatted at 2026-08-10 06:21:08 on dist-test-slave-q9h9"
server_key: "cb8a09b5354eb6fdb2464449be5bbde2"
server_key_iv: "82e33568d082d4e068764ae0cc16cb1c"
server_key_version: "encryptionkey@0"
I20260810 06:21:08.912176 17644 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260810 06:21:08.914326 17669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:08.915386 17644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:21:08.915478 17644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "202121fcadfb44d99070dc9094407551"
format_stamp: "Formatted at 2026-08-10 06:21:08 on dist-test-slave-q9h9"
server_key: "cb8a09b5354eb6fdb2464449be5bbde2"
server_key_iv: "82e33568d082d4e068764ae0cc16cb1c"
server_key_version: "encryptionkey@0"
I20260810 06:21:08.915566 17644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:21:08.928579 17644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:21:08.931473 17644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:21:08.931720 17644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:21:08.932260 17644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:21:08.932997 17644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:21:08.933041 17644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:08.933080 17644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:21:08.933108 17644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:08.941200 17644 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:38047
I20260810 06:21:08.941268 17856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:38047 every 8 connection(s)
I20260810 06:21:08.942096 17644 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:21:08.950234 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 17644
I20260810 06:21:08.950356 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
Aug 10 06:21:08 dist-test-slave-q9h9 krb5kdc[17000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342868, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:21:08.950660 32563 external_mini_cluster.cc:1428] Setting key e1a0239f1f649cd7986c6e63947197c8
I20260810 06:21:08.954361 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:08.943762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:49053 (local address 127.31.204.254:40559)
0810 06:21:08.943892 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:08.943896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:08.944622 (+ 726us) server_negotiation.cc:407] Connection header received
0810 06:21:08.945861 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:08.945864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:08.945911 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:08.945979 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:08.947032 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.947557 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.948578 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.948777 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.948840 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:08.951154 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:21:08.951158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:21:08.951160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:21:08.951186 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:21:08.952485 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:08.952948 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:08.952952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:08.952953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:08.953032 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:08.953344 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:08.953347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:08.953349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:08.953530 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:21:08.953615 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:08.954085 (+ 470us) server_negotiation.cc:299] Negotiation successful
0810 06:21:08.954203 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:21:08.955065 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:08.944050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:40559 (local address 127.31.204.195:49053)
0810 06:21:08.944469 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:08.944509 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:08.945677 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:08.946152 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:08.946158 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:08.946490 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:08.946921 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:08.946933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.947647 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.947649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:08.948490 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:08.948499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.948876 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.948879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:08.949004 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:08.949462 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:08.949476 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:08.950990 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:21:08.952594 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:21:08.952605 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:21:08.952617 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:21:08.952857 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:21:08.953119 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:21:08.953122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:21:08.953124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:21:08.953257 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:21:08.953626 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:21:08.953633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:21:08.953851 (+ 218us) client_negotiation.cc:769] Sending connection context
0810 06:21:08.954006 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 06:21:08.954321 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
I20260810 06:21:08.955981 17857 heartbeater.cc:347] Connected to a master server at 127.31.204.254:40559
I20260810 06:21:08.956166 17857 heartbeater.cc:464] Registering TS with master...
I20260810 06:21:08.956568 17857 heartbeater.cc:511] Master 127.31.204.254:40559 requested a full tablet report, sending...
I20260810 06:21:08.957432 17090 ts_manager.cc:194] Registered new tserver with Master: 202121fcadfb44d99070dc9094407551 (127.31.204.195:38047)
I20260810 06:21:08.957958 17090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:49053
I20260810 06:21:08.963975 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:21:08.971838 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:08.965704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38562 (local address 127.31.204.254:40559)
0810 06:21:08.965829 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:08.965833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:08.965964 (+ 131us) server_negotiation.cc:407] Connection header received
0810 06:21:08.966091 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:08.966093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:08.966129 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:08.966190 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:08.966816 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.967283 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.968040 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.968239 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.968301 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:08.969448 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:21:08.969451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:21:08.969453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:21:08.969474 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:21:08.970608 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:08.970944 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:08.970946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:08.970947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:08.971008 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:08.971204 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:08.971206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:08.971206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:08.971386 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:21:08.971478 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:08.971623 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 06:21:08.971703 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:21:08.974181 17090 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:21:08.976938 17090 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:21:08.990221 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:08.984683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:41301 (local address 127.0.0.1:51932)
0810 06:21:08.985215 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:08.985273 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:08.985393 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:08.986145 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:08.986147 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:08.986166 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:21:08.986280 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:08.986288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.987641 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.987644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:08.988740 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:08.988748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.989860 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.989863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:08.989948 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:08.989968 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 06:21:08.990060 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 06:21:08.990117 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":26,"spinlock_wait_cycles":9344,"thread_start_us":79,"threads_started":1}
I20260810 06:21:08.990247 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:08.985527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.31.204.195:38047)
0810 06:21:08.986257 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:08.986271 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:08.986288 (+ 17us) server_negotiation.cc:407] Connection header received
0810 06:21:08.986339 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:08.986344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:08.986517 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:08.986663 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:21:08.986875 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.987821 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.989039 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.989556 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.989664 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:08.989783 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:08.989970 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 06:21:08.990100 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":141,"threads_started":1}
I20260810 06:21:08.990345 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:08.984514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:40539 (local address 127.0.0.1:41030)
0810 06:21:08.985200 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:08.985246 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:08.985392 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:08.986101 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:08.986107 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:08.986130 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:21:08.986356 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:08.986364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.987401 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.987404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:08.988679 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:08.988690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.989931 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.989935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:08.990043 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:08.990066 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 06:21:08.990131 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 06:21:08.990199 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":100,"spinlock_wait_cycles":9728,"thread_start_us":63,"threads_started":1}
I20260810 06:21:08.990208 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:08.985181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:38047 (local address 127.0.0.1:34398)
0810 06:21:08.985915 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:08.985927 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:08.986003 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:08.986657 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:08.986660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:08.986690 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:21:08.986772 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:08.986778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.987935 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.987938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:08.988930 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:08.988940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.989671 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.989674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:08.989796 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:08.989836 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 06:21:08.989936 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 06:21:08.990068 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260810 06:21:08.990805 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:08.985186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51932 (local address 127.31.204.193:41301)
0810 06:21:08.985634 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:08.985645 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:08.985664 (+ 19us) server_negotiation.cc:407] Connection header received
0810 06:21:08.985725 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:08.985729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:08.985869 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:08.985981 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:21:08.986402 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.987511 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.988853 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.989631 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.989747 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:08.989864 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:08.990566 (+ 702us) server_negotiation.cc:299] Negotiation successful
0810 06:21:08.990666 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":153,"threads_started":1}
I20260810 06:21:08.990958 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:08.984661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.31.204.194:40539)
0810 06:21:08.984993 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:08.985005 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:08.985523 (+ 518us) server_negotiation.cc:407] Connection header received
0810 06:21:08.985602 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:08.985607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:08.985809 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:08.985955 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:21:08.986445 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.987284 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.988832 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:08.989632 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:08.989747 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:08.989864 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:08.990717 (+ 853us) server_negotiation.cc:299] Negotiation successful
0810 06:21:08.990815 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":130,"threads_started":1}
I20260810 06:21:08.993100 17755 tablet_service.cc:1467] Processing CreateTablet for tablet eabf4e552e4047e9896f26c4bae60fe4 (DEFAULT_TABLE table=test-table [id=ab32ca71882645a6b61a0180d23ee2aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:21:08.993948 17755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eabf4e552e4047e9896f26c4bae60fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:21:08.993659 17526 tablet_service.cc:1467] Processing CreateTablet for tablet eabf4e552e4047e9896f26c4bae60fe4 (DEFAULT_TABLE table=test-table [id=ab32ca71882645a6b61a0180d23ee2aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:21:08.993659 17302 tablet_service.cc:1467] Processing CreateTablet for tablet eabf4e552e4047e9896f26c4bae60fe4 (DEFAULT_TABLE table=test-table [id=ab32ca71882645a6b61a0180d23ee2aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:21:08.994719 17526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eabf4e552e4047e9896f26c4bae60fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:21:08.994719 17302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eabf4e552e4047e9896f26c4bae60fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:21:09.001014 17891 tablet_bootstrap.cc:492] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551: Bootstrap starting.
I20260810 06:21:09.001448 17892 tablet_bootstrap.cc:492] T eabf4e552e4047e9896f26c4bae60fe4 P 77c7a4804d7346a6b12c412324498deb: Bootstrap starting.
I20260810 06:21:09.002229 17893 tablet_bootstrap.cc:492] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f: Bootstrap starting.
I20260810 06:21:09.003330 17891 tablet_bootstrap.cc:654] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551: Neither blocks nor log segments found. Creating new log.
I20260810 06:21:09.003968 17892 tablet_bootstrap.cc:654] T eabf4e552e4047e9896f26c4bae60fe4 P 77c7a4804d7346a6b12c412324498deb: Neither blocks nor log segments found. Creating new log.
I20260810 06:21:09.004257 17893 tablet_bootstrap.cc:654] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f: Neither blocks nor log segments found. Creating new log.
I20260810 06:21:09.004613 17891 log.cc:824] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551: Log is configured to *not* fsync() on all Append() calls
I20260810 06:21:09.004678 17892 log.cc:824] T eabf4e552e4047e9896f26c4bae60fe4 P 77c7a4804d7346a6b12c412324498deb: Log is configured to *not* fsync() on all Append() calls
I20260810 06:21:09.004933 17893 log.cc:824] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f: Log is configured to *not* fsync() on all Append() calls
I20260810 06:21:09.006794 17892 tablet_bootstrap.cc:492] T eabf4e552e4047e9896f26c4bae60fe4 P 77c7a4804d7346a6b12c412324498deb: No bootstrap required, opened a new log
I20260810 06:21:09.006790 17893 tablet_bootstrap.cc:492] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f: No bootstrap required, opened a new log
I20260810 06:21:09.006891 17891 tablet_bootstrap.cc:492] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551: No bootstrap required, opened a new log
I20260810 06:21:09.006980 17893 ts_tablet_manager.cc:1397] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:21:09.006980 17892 ts_tablet_manager.cc:1397] T eabf4e552e4047e9896f26c4bae60fe4 P 77c7a4804d7346a6b12c412324498deb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:21:09.007072 17891 ts_tablet_manager.cc:1397] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260810 06:21:09.009661 17891 raft_consensus.cc:348] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6670ad5d996949269a878c05e77e0f2f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 40539 } } peers { permanent_uuid: "77c7a4804d7346a6b12c412324498deb" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41301 } } peers { permanent_uuid: "202121fcadfb44d99070dc9094407551" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38047 } }
I20260810 06:21:09.009865 17891 raft_consensus.cc:374] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:21:09.009945 17891 raft_consensus.cc:729] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202121fcadfb44d99070dc9094407551, State: Initialized, Role: FOLLOWER
I20260810 06:21:09.010354 17891 consensus_queue.cc:260] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6670ad5d996949269a878c05e77e0f2f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 40539 } } peers { permanent_uuid: "77c7a4804d7346a6b12c412324498deb" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41301 } } peers { permanent_uuid: "202121fcadfb44d99070dc9094407551" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38047 } }
I20260810 06:21:09.010638 17893 raft_consensus.cc:348] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6670ad5d996949269a878c05e77e0f2f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 40539 } } peers { permanent_uuid: "77c7a4804d7346a6b12c412324498deb" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41301 } } peers { permanent_uuid: "202121fcadfb44d99070dc9094407551" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38047 } }
I20260810 06:21:09.010638 17892 raft_consensus.cc:348] T eabf4e552e4047e9896f26c4bae60fe4 P 77c7a4804d7346a6b12c412324498deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6670ad5d996949269a878c05e77e0f2f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 40539 } } peers { permanent_uuid: "77c7a4804d7346a6b12c412324498deb" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41301 } } peers { permanent_uuid: "202121fcadfb44d99070dc9094407551" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38047 } }
I20260810 06:21:09.010932 17893 raft_consensus.cc:374] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:21:09.010951 17892 raft_consensus.cc:374] T eabf4e552e4047e9896f26c4bae60fe4 P 77c7a4804d7346a6b12c412324498deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:21:09.010967 17857 heartbeater.cc:503] Master 127.31.204.254:40559 was elected leader, sending a full tablet report...
I20260810 06:21:09.011026 17892 raft_consensus.cc:729] T eabf4e552e4047e9896f26c4bae60fe4 P 77c7a4804d7346a6b12c412324498deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c7a4804d7346a6b12c412324498deb, State: Initialized, Role: FOLLOWER
I20260810 06:21:09.011023 17893 raft_consensus.cc:729] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6670ad5d996949269a878c05e77e0f2f, State: Initialized, Role: FOLLOWER
I20260810 06:21:09.011318 17891 ts_tablet_manager.cc:1428] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260810 06:21:09.011442 17893 consensus_queue.cc:260] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6670ad5d996949269a878c05e77e0f2f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 40539 } } peers { permanent_uuid: "77c7a4804d7346a6b12c412324498deb" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41301 } } peers { permanent_uuid: "202121fcadfb44d99070dc9094407551" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38047 } }
I20260810 06:21:09.011519 17892 consensus_queue.cc:260] T eabf4e552e4047e9896f26c4bae60fe4 P 77c7a4804d7346a6b12c412324498deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6670ad5d996949269a878c05e77e0f2f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 40539 } } peers { permanent_uuid: "77c7a4804d7346a6b12c412324498deb" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41301 } } peers { permanent_uuid: "202121fcadfb44d99070dc9094407551" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38047 } }
I20260810 06:21:09.012228 17638 heartbeater.cc:503] Master 127.31.204.254:40559 was elected leader, sending a full tablet report...
I20260810 06:21:09.012486 17893 ts_tablet_manager.cc:1428] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 06:21:09.012486 17892 ts_tablet_manager.cc:1428] T eabf4e552e4047e9896f26c4bae60fe4 P 77c7a4804d7346a6b12c412324498deb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 06:21:09.012816 17407 heartbeater.cc:503] Master 127.31.204.254:40559 was elected leader, sending a full tablet report...
W20260810 06:21:09.041069 17408 tablet.cc:2369] T eabf4e552e4047e9896f26c4bae60fe4 P 77c7a4804d7346a6b12c412324498deb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:21:09.192971 17861 tablet.cc:2369] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:21:09.247560 17639 tablet.cc:2369] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:21:09.353902 17904 raft_consensus.cc:482] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:21:09.354140 17904 raft_consensus.cc:504] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6670ad5d996949269a878c05e77e0f2f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 40539 } } peers { permanent_uuid: "77c7a4804d7346a6b12c412324498deb" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41301 } } peers { permanent_uuid: "202121fcadfb44d99070dc9094407551" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38047 } }
I20260810 06:21:09.355257 17904 leader_election.cc:288] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6670ad5d996949269a878c05e77e0f2f (127.31.204.194:40539), 77c7a4804d7346a6b12c412324498deb (127.31.204.193:41301)
I20260810 06:21:09.359963 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:09.355583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:56629 (local address 127.31.204.194:40539)
0810 06:21:09.355690 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:09.355693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:09.355708 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:21:09.355883 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:09.355886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:09.355934 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:09.356011 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:21:09.356265 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:09.357027 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:09.358706 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:09.359330 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:09.359441 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:09.359490 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:09.359769 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 06:21:09.359837 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:21:09.359963 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:09.355633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:34861 (local address 127.31.204.193:41301)
0810 06:21:09.355757 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:09.355762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:09.355980 (+ 218us) server_negotiation.cc:407] Connection header received
0810 06:21:09.356064 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:09.356067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:09.356117 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:09.356173 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:21:09.356407 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:09.357104 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:09.358698 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:09.359330 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:09.359441 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:09.359490 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:09.359769 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 06:21:09.359837 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:21:09.360013 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:09.355438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:40539 (local address 127.31.204.195:56629)
0810 06:21:09.355639 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:09.355654 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:09.355744 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:09.356033 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:09.356037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:09.356052 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:21:09.356152 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:09.356157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:09.357123 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:09.357125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:09.358584 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:09.358594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:09.359452 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:09.359456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:09.359609 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:09.359669 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 06:21:09.359749 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 06:21:09.359859 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":191}
I20260810 06:21:09.360023 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:09.355523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:41301 (local address 127.31.204.195:34861)
0810 06:21:09.355900 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:09.355910 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:09.355987 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:09.356197 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:09.356200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:09.356216 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:21:09.356322 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:09.356327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:09.357188 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:09.357191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:09.358583 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:09.358594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:09.359452 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:09.359456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:09.359624 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:09.359669 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 06:21:09.359749 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 06:21:09.359865 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":104,"thread_start_us":111,"threads_started":1}
I20260810 06:21:09.360656 17565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eabf4e552e4047e9896f26c4bae60fe4" candidate_uuid: "202121fcadfb44d99070dc9094407551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6670ad5d996949269a878c05e77e0f2f" is_pre_election: true
I20260810 06:21:09.360716 17338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eabf4e552e4047e9896f26c4bae60fe4" candidate_uuid: "202121fcadfb44d99070dc9094407551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c7a4804d7346a6b12c412324498deb" is_pre_election: true
I20260810 06:21:09.360921 17565 raft_consensus.cc:2454] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 202121fcadfb44d99070dc9094407551 in term 0.
I20260810 06:21:09.360981 17338 raft_consensus.cc:2454] T eabf4e552e4047e9896f26c4bae60fe4 P 77c7a4804d7346a6b12c412324498deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 202121fcadfb44d99070dc9094407551 in term 0.
I20260810 06:21:09.361392 17670 leader_election.cc:302] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202121fcadfb44d99070dc9094407551, 6670ad5d996949269a878c05e77e0f2f; no voters:
I20260810 06:21:09.361632 17904 raft_consensus.cc:2781] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:21:09.361714 17904 raft_consensus.cc:482] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:21:09.361745 17904 raft_consensus.cc:3037] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:21:09.362809 17904 raft_consensus.cc:504] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6670ad5d996949269a878c05e77e0f2f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 40539 } } peers { permanent_uuid: "77c7a4804d7346a6b12c412324498deb" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41301 } } peers { permanent_uuid: "202121fcadfb44d99070dc9094407551" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38047 } }
I20260810 06:21:09.363123 17904 leader_election.cc:288] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [CANDIDATE]: Term 1 election: Requested vote from peers 6670ad5d996949269a878c05e77e0f2f (127.31.204.194:40539), 77c7a4804d7346a6b12c412324498deb (127.31.204.193:41301)
I20260810 06:21:09.363479 17565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eabf4e552e4047e9896f26c4bae60fe4" candidate_uuid: "202121fcadfb44d99070dc9094407551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6670ad5d996949269a878c05e77e0f2f"
I20260810 06:21:09.363484 17338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eabf4e552e4047e9896f26c4bae60fe4" candidate_uuid: "202121fcadfb44d99070dc9094407551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c7a4804d7346a6b12c412324498deb"
I20260810 06:21:09.363581 17565 raft_consensus.cc:3037] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:21:09.363587 17338 raft_consensus.cc:3037] T eabf4e552e4047e9896f26c4bae60fe4 P 77c7a4804d7346a6b12c412324498deb [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:21:09.364601 17338 raft_consensus.cc:2454] T eabf4e552e4047e9896f26c4bae60fe4 P 77c7a4804d7346a6b12c412324498deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 202121fcadfb44d99070dc9094407551 in term 1.
I20260810 06:21:09.364619 17565 raft_consensus.cc:2454] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 202121fcadfb44d99070dc9094407551 in term 1.
I20260810 06:21:09.364856 17670 leader_election.cc:302] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202121fcadfb44d99070dc9094407551, 6670ad5d996949269a878c05e77e0f2f; no voters:
I20260810 06:21:09.365018 17904 raft_consensus.cc:2781] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:21:09.365252 17904 raft_consensus.cc:686] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [term 1 LEADER]: Becoming Leader. State: Replica: 202121fcadfb44d99070dc9094407551, State: Running, Role: LEADER
I20260810 06:21:09.365540 17904 consensus_queue.cc:237] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6670ad5d996949269a878c05e77e0f2f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 40539 } } peers { permanent_uuid: "77c7a4804d7346a6b12c412324498deb" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41301 } } peers { permanent_uuid: "202121fcadfb44d99070dc9094407551" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38047 } }
I20260810 06:21:09.368129 17090 catalog_manager.cc:5495] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 reported cstate change: term changed from 0 to 1, leader changed from <none> to 202121fcadfb44d99070dc9094407551 (127.31.204.195). New cstate: current_term: 1 leader_uuid: "202121fcadfb44d99070dc9094407551" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6670ad5d996949269a878c05e77e0f2f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 40539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77c7a4804d7346a6b12c412324498deb" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "202121fcadfb44d99070dc9094407551" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38047 } health_report { overall_health: HEALTHY } } }
I20260810 06:21:09.436218 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:09.433072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.31.204.195:38047)
0810 06:21:09.433189 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:09.433193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:09.433207 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:21:09.433280 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:09.433282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:09.433339 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:09.433434 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:21:09.433627 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:09.434150 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:09.434945 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:09.435156 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:09.435251 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:09.435480 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:21:09.435870 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:21:09.435966 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:09.435999 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 06:21:09.436048 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 06:21:09.441839 17338 raft_consensus.cc:1261] T eabf4e552e4047e9896f26c4bae60fe4 P 77c7a4804d7346a6b12c412324498deb [term 1 FOLLOWER]: Refusing update from remote peer 202121fcadfb44d99070dc9094407551: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:21:09.441857 17565 raft_consensus.cc:1261] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f [term 1 FOLLOWER]: Refusing update from remote peer 202121fcadfb44d99070dc9094407551: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:21:09.442553 17904 consensus_queue.cc:1035] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6670ad5d996949269a878c05e77e0f2f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 40539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:21:09.442730 17910 consensus_queue.cc:1035] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77c7a4804d7346a6b12c412324498deb" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:21:09.450829 17920 mvcc.cc:205] Tried to move back new op lower bound from 7316860393226842112 to 7316860392923668480. Current Snapshot: MvccSnapshot[applied={T|T < 7316860393226842112}]
I20260810 06:21:09.453086 17917 mvcc.cc:205] Tried to move back new op lower bound from 7316860393226842112 to 7316860392923668480. Current Snapshot: MvccSnapshot[applied={T|T < 7316860393226842112 or (T in {7316860393226842112})}]
I20260810 06:21:09.458591 17078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:21:09.458755 17078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:21:09.460893 17078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc060fa01a2c45468d883f7fc2336ea5: Sending DeleteTablet for 3 replicas of tablet eabf4e552e4047e9896f26c4bae60fe4
I20260810 06:21:09.461536 17526 tablet_service.cc:1514] Processing DeleteTablet for tablet eabf4e552e4047e9896f26c4bae60fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:21:09 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:41030
I20260810 06:21:09.461735 17302 tablet_service.cc:1514] Processing DeleteTablet for tablet eabf4e552e4047e9896f26c4bae60fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:21:09 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:51932
I20260810 06:21:09.462132 17755 tablet_service.cc:1514] Processing DeleteTablet for tablet eabf4e552e4047e9896f26c4bae60fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:21:09 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:34398
I20260810 06:21:09.462220 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 17192
I20260810 06:21:09.476635 17939 tablet_replica.cc:330] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551: stopping tablet replica
I20260810 06:21:09.476924 17939 raft_consensus.cc:2229] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:21:09.477250 17939 raft_consensus.cc:2258] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:21:09.477591 17937 tablet_replica.cc:330] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f: stopping tablet replica
I20260810 06:21:09.477842 17937 raft_consensus.cc:2229] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:21:09.478132 17937 raft_consensus.cc:2258] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:21:09.480448 17939 ts_tablet_manager.cc:1905] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:21:09.482966 17937 ts_tablet_manager.cc:1905] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:21:09.483088 17939 ts_tablet_manager.cc:1918] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:21:09.483253 17939 log.cc:1196] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/eabf4e552e4047e9896f26c4bae60fe4
I20260810 06:21:09.483618 17939 ts_tablet_manager.cc:1939] T eabf4e552e4047e9896f26c4bae60fe4 P 202121fcadfb44d99070dc9094407551: Deleting consensus metadata
I20260810 06:21:09.485415 17937 ts_tablet_manager.cc:1918] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:21:09.485415 17061 catalog_manager.cc:4841] TS 202121fcadfb44d99070dc9094407551 (127.31.204.195:38047): tablet eabf4e552e4047e9896f26c4bae60fe4 (table test-table [id=ab32ca71882645a6b61a0180d23ee2aa]) successfully deleted
I20260810 06:21:09.485503 17937 log.cc:1196] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/eabf4e552e4047e9896f26c4bae60fe4
I20260810 06:21:09.485767 17937 ts_tablet_manager.cc:1939] T eabf4e552e4047e9896f26c4bae60fe4 P 6670ad5d996949269a878c05e77e0f2f: Deleting consensus metadata
I20260810 06:21:09.486815 17051 catalog_manager.cc:4841] TS 6670ad5d996949269a878c05e77e0f2f (127.31.204.194:40539): tablet eabf4e552e4047e9896f26c4bae60fe4 (table test-table [id=ab32ca71882645a6b61a0180d23ee2aa]) successfully deleted
W20260810 06:21:09.491799 17061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.204.193:41301 (error 108)
I20260810 06:21:09.492653 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 17415
I20260810 06:21:09.524919 17943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:09.492532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:41301 (local address 127.0.0.1:51948)
0810 06:21:09.524693 (+ 32161us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:09.524799 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:41301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":57,"threads_started":1}
W20260810 06:21:09.525266 17061 catalog_manager.cc:4584] TS 77c7a4804d7346a6b12c412324498deb (127.31.204.193:41301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eabf4e552e4047e9896f26c4bae60fe4: Network error: Client connection negotiation failed: client connection to 127.31.204.193:41301: connect: Connection refused (error 111)
I20260810 06:21:09.529686 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 17644
I20260810 06:21:09.538784 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 17019
2026-08-10T06:21:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2699 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:21:09 dist-test-slave-q9h9 krb5kdc[17947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:21:09 dist-test-slave-q9h9 krb5kdc[17947](info): set up 2 sockets
Aug 10 06:21:09 dist-test-slave-q9h9 krb5kdc[17947](info): commencing operation
krb5kdc: starting...
W20260810 06:21:10.703482 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:21:10 dist-test-slave-q9h9 krb5kdc[17947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:21:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:21:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.204.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.204.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.204.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/rangeradmin_127.31.204.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.204.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/rangeradmin_127.31.204.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.204.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.204.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.204.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/rangerlookup_127.31.204.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.204.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/rangerlookup_127.31.204.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.204.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.204.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.204.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/HTTP_127.31.204.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.204.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/HTTP_127.31.204.212@KRBTEST.COM.keytab.
I20260810 06:21:10.849725 32563 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7DS5L1/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/postgres -l logfile start
2026-08-10 06:21:12.932 UTC [17994] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 06:21:12.932 UTC [17994] LOG: listening on IPv4 address "127.31.204.212", port 50885
2026-08-10 06:21:12.935 UTC [17994] LOG: listening on Unix socket "/tmp/.s.PGSQL.50885"
2026-08-10 06:21:12.955 UTC [17998] LOG: database system was shut down at 2026-08-10 06:21:11 UTC
2026-08-10 06:21:12.958 UTC [17994] LOG: database system is ready to accept connections
I20260810 06:21:14.013877 32563 mini_postgres.cc:96] Postgres bound to 50885
2026-08-10 06:21:14.022 UTC [18012] FATAL: database "slave" does not exist
127.31.204.212:50885 - accepting connections
I20260810 06:21:14.023279 32563 mini_ranger.cc:161] Starting Ranger...
I20260810 06:21:14.038591 32563 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 06:21:14.371518 32563 mini_ranger.cc:87] Created ranger Postgres database
I20260810 06:21:14.371626 32563 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ranger-admin
2026-08-10 06:21:14,757 [I] DB FLAVOR :POSTGRES
2026-08-10 06:21:14,757 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 06:21:14,757 [I] Checking connection..
2026-08-10 06:21:14,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 06:21:15,021 [I] Checking connection passed.
2026-08-10 06:21:15,021 [I] --------- Verifying version history table ---------
2026-08-10 06:21:15,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 06:21:15,271 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 06:21:15,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 06:21:15,498 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 06:21:15,498 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 06:21:15,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 06:21:15.716 UTC [18113] WARNING: there is no transaction in progress
2026-08-10 06:21:15,722 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 06:21:15,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 06:21:15,941 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 06:21:15,942 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 06:21:15,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 06:21:16,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 06:21:16,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q9h9.c.gcp-upstream.internal','N') ;"
2026-08-10 06:21:16,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 06:21:16,881 [I] Table x_portal_user does not exist in database ranger
2026-08-10 06:21:16,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 06:21:17,137 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 06:21:17,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 06:21:17,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 06:21:17,612 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 06:21:17,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 06:21:18.444 UTC [18300] WARNING: there is no transaction in progress
2026-08-10 06:21:18.458 UTC [18300] WARNING: there is no transaction in progress
2026-08-10 06:21:18.471 UTC [18300] WARNING: there is no transaction in progress
2026-08-10 06:21:18.484 UTC [18300] WARNING: there is no transaction in progress
2026-08-10 06:21:18.497 UTC [18300] WARNING: there is no transaction in progress
2026-08-10 06:21:18.513 UTC [18300] WARNING: there is no transaction in progress
2026-08-10 06:21:18.561 UTC [18300] WARNING: there is no transaction in progress
2026-08-10 06:21:18.567 UTC [18300] WARNING: there is no transaction in progress
2026-08-10 06:21:18.579 UTC [18300] WARNING: there is no transaction in progress
2026-08-10 06:21:18.588 UTC [18300] WARNING: there is no transaction in progress
2026-08-10 06:21:18.597 UTC [18300] WARNING: there is no transaction in progress
2026-08-10 06:21:18.609 UTC [18300] WARNING: there is no transaction in progress
2026-08-10 06:21:18.619 UTC [18300] WARNING: there is no transaction in progress
2026-08-10 06:21:18.624 UTC [18300] WARNING: there is no transaction in progress
2026-08-10 06:21:18.630 UTC [18300] WARNING: there is no transaction in progress
2026-08-10 06:21:18.635 UTC [18300] WARNING: there is no transaction in progress
2026-08-10 06:21:19,129 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 06:21:19,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 06:21:19,370 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 06:21:19,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 06:21:19,613 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 06:21:19,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 06:21:19,869 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 06:21:19,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 06:21:20,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 06:21:20,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q9h9.c.gcp-upstream.internal';"
2026-08-10 06:21:20,565 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 06:21:20,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 06:21:20,809 [I] DB_PATCHES have already been applied
I20260810 06:21:20.814977 32563 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ranger-admin:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7DS5L1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 06:21:20.815047 32563 mini_ranger.cc:193] Using host: 127.31.204.212
I20260810 06:21:20.819561 32563 mini_ranger.cc:227] Ranger admin URL: http://127.31.204.212:57351
Aug 10, 2026 6:21:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:21:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:21:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 6:21:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 6:21:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 6:21:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 6:21:21 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 06:21:22.102574 32563 mini_ranger.cc:160] Time spent starting Ranger: real 8.079s user 0.004s sys 0.008s
Aug 10, 2026 6:21:41 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 6:21:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Aug 10, 2026 6:21:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 6:21:41 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 6:21:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 6:21:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 6:21:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 6:21:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 6:21:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 6:21:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 6:21:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 6:21:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 6:21:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 6:21:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 6:21:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 6:21:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 6:21:41 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 6:21:42 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 06:21:43.148861 32563 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.204.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.204.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.204.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/rangerkms_127.31.204.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.204.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/rangerkms_127.31.204.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.204.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.204.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.204.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/HTTP_127.31.204.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.204.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/HTTP_127.31.204.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 06:21:43 dist-test-slave-q9h9 krb5kdc[17947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342903, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 06:21:43.223217 32563 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 06:21:43.232528 32563 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 06:21:43.859961 32563 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 06:21:43.860050 32563 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ranger-kms
I20260810 06:21:43.860169 32563 mini_ranger_kms.cc:227] Using postgres at 127.31.204.212:50885
2026-08-10 06:21:44,364 [I] DB FLAVOR :POSTGRES
2026-08-10 06:21:44,365 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 06:21:44,365 [I] Checking connection
2026-08-10 06:21:44,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 06:21:44,614 [I] connection success
2026-08-10 06:21:44,615 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 06:21:44,615 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 06:21:44,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 06:21:44,861 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 06:21:44,861 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 06:21:44,861 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 06:21:44,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:50885/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 06:21:45,100 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 06:21:45.538054 32563 mini_ranger_kms.cc:310] Created kms service
I20260810 06:21:45.767140 32563 mini_ranger_kms.cc:326] Created kudu user
I20260810 06:21:45.809813 32563 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 06:21:45 dist-test-slave-q9h9 krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342903, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
I20260810 06:21:46.192055 32563 mini_ranger_kms.cc:384] Added ranger policy
I20260810 06:21:46.192220 32563 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ranger-kms:/tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 06:21:46.192240 32563 mini_ranger_kms.cc:240] Using host: 127.31.204.212
I20260810 06:21:46.196839 32563 mini_ranger_kms.cc:276] Ranger KMS PID: 18580
I20260810 06:21:46.196998 32563 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.204.212:35787
Aug 10, 2026 6:21:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:21:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:21:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 6:21:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 6:21:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 6:21:47 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 06:21:48 dist-test-slave-q9h9 krb5kdc[17947](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786342908, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.204.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:21:48.788422 32563 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.565s user 0.016s sys 0.244s
I20260810 06:21:48.788582 32563 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 06:21:48.788728 32563 mini_ranger_kms.cc:401] 127.31.204.212:35787/kms/v1/keys
Aug 10 06:21:48 dist-test-slave-q9h9 krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342908, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.204.212@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
Aug 10 06:21:49 dist-test-slave-q9h9 krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342908, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.204.212@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
Aug 10 06:21:51 dist-test-slave-q9h9 krb5kdc[17947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:21:51.533600 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:34795
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:41771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:34795
--ranger_config_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.204.212:35787/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:21:51.669984 18657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:21:51.670279 18657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:21:51.670320 18657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:21:51.674340 18657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:21:51.674397 18657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:21:51.674415 18657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:21:51.674430 18657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:21:51.674448 18657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:21:51.679795 18657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:41771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.204.212:35787/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:34795
--ranger_config_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:34795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:21:51.680787 18657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:21:51.681777 18657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:21:51.690369 18666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:51.690542 18671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:51.690389 18668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:51.690900 18657 server_base.cc:1034] running on GCE node
I20260810 06:21:51.691433 18657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:21:51.692471 18657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:21:51.693636 18657 hybrid_clock.cc:648] HybridClock initialized: now 1786342911693620 us; error 44 us; skew 500 ppm
Aug 10 06:21:51 dist-test-slave-q9h9 krb5kdc[17947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342911, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:21:51.701171 18657 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:21:51.702235 18657 webserver.cc:460] Webserver started at http://127.31.204.254:34427/ using document root <none> and password file <none>
I20260810 06:21:51.702919 18657 fs_manager.cc:359] Metadata directory not provided
I20260810 06:21:51.703003 18657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:21:51.703241 18657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 06:21:51 dist-test-slave-q9h9 krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342911, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
I20260810 06:21:51.750427 18657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "e240096bf3954611930ba59088672567"
format_stamp: "Formatted at 2026-08-10 06:21:51 on dist-test-slave-q9h9"
server_key: "f075c6ec10e8069d8eb9a0cd0d364ea6"
server_key_iv: "1e8faa2fe1bd7ffb34e4dca124126ad7"
server_key_version: "kuduclusterkey@0"
I20260810 06:21:51.751204 18657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "e240096bf3954611930ba59088672567"
format_stamp: "Formatted at 2026-08-10 06:21:51 on dist-test-slave-q9h9"
server_key: "f075c6ec10e8069d8eb9a0cd0d364ea6"
server_key_iv: "1e8faa2fe1bd7ffb34e4dca124126ad7"
server_key_version: "kuduclusterkey@0"
I20260810 06:21:51.756352 18657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 06:21:51.773689 18680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:51.775332 18657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 06:21:51.775522 18657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "e240096bf3954611930ba59088672567"
format_stamp: "Formatted at 2026-08-10 06:21:51 on dist-test-slave-q9h9"
server_key: "f075c6ec10e8069d8eb9a0cd0d364ea6"
server_key_iv: "1e8faa2fe1bd7ffb34e4dca124126ad7"
server_key_version: "kuduclusterkey@0"
I20260810 06:21:51.775698 18657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:21:51.801028 18657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:21:51.803478 18657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:21:51.803656 18657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:21:51.810581 18657 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:34795
I20260810 06:21:51.810595 18763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:34795 every 8 connection(s)
I20260810 06:21:51.811503 18657 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:21:51.814148 18764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:21:51.819358 18764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567: Bootstrap starting.
I20260810 06:21:51.821640 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 18657
I20260810 06:21:51.821658 18764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567: Neither blocks nor log segments found. Creating new log.
I20260810 06:21:51.821771 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:21:51.822119 32563 external_mini_cluster.cc:1428] Setting key da5fecc63ac22cb7a4938ae7271c648c
I20260810 06:21:51.822542 18764 log.cc:824] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567: Log is configured to *not* fsync() on all Append() calls
I20260810 06:21:51.824749 18764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567: No bootstrap required, opened a new log
I20260810 06:21:51.828809 18764 raft_consensus.cc:348] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e240096bf3954611930ba59088672567" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34795 } }
I20260810 06:21:51.829069 18764 raft_consensus.cc:374] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:21:51.829130 18764 raft_consensus.cc:729] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e240096bf3954611930ba59088672567, State: Initialized, Role: FOLLOWER
I20260810 06:21:51.829547 18764 consensus_queue.cc:260] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e240096bf3954611930ba59088672567" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34795 } }
I20260810 06:21:51.829686 18764 raft_consensus.cc:388] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:21:51.829749 18764 raft_consensus.cc:482] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:21:51.829840 18764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 06:21:51 dist-test-slave-q9h9 krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:21:51.830854 18764 raft_consensus.cc:504] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e240096bf3954611930ba59088672567" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34795 } }
I20260810 06:21:51.831166 18764 leader_election.cc:302] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e240096bf3954611930ba59088672567; no voters:
I20260810 06:21:51.831472 18764 leader_election.cc:288] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:21:51.831539 18771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:21:51.831818 18771 raft_consensus.cc:686] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567 [term 1 LEADER]: Becoming Leader. State: Replica: e240096bf3954611930ba59088672567, State: Running, Role: LEADER
I20260810 06:21:51.832160 18771 consensus_queue.cc:237] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e240096bf3954611930ba59088672567" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34795 } }
I20260810 06:21:51.832749 18764 sys_catalog.cc:564] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:21:51.833702 18772 sys_catalog.cc:455] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e240096bf3954611930ba59088672567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e240096bf3954611930ba59088672567" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34795 } } }
I20260810 06:21:51.834117 18772 sys_catalog.cc:458] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567 [sys.catalog]: This master's current role is: LEADER
I20260810 06:21:51.833997 18773 sys_catalog.cc:455] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e240096bf3954611930ba59088672567. Latest consensus state: current_term: 1 leader_uuid: "e240096bf3954611930ba59088672567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e240096bf3954611930ba59088672567" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34795 } } }
I20260810 06:21:51.834424 18773 sys_catalog.cc:458] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567 [sys.catalog]: This master's current role is: LEADER
I20260810 06:21:51.835229 18784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:21:51.835604 18764 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 06:21:51.836483 18770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:51.823160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.31.204.254:34795)
0810 06:21:51.823564 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:51.823582 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:51.823624 (+ 42us) server_negotiation.cc:407] Connection header received
0810 06:21:51.824412 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:51.824434 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:51.824815 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:51.825169 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:51.825813 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:51.827323 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:51.828120 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:51.828787 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:51.828942 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:51.831344 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:21:51.831351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:21:51.831366 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:21:51.831399 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:21:51.833333 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:51.833859 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:51.833865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:51.833870 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:51.834005 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:51.834371 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:51.834374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:51.834376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:51.834908 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:21:51.835100 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:51.835469 (+ 369us) server_negotiation.cc:299] Negotiation successful
0810 06:21:51.835744 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":111,"threads_started":1}
I20260810 06:21:51.838151 18784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:21:51.847389 18784 catalog_manager.cc:1347] Generated new cluster ID: 08cf592f29da41aead7f397b6aa8be5f
I20260810 06:21:51.847456 18784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:21:51.882071 18784 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:21:51.882974 18784 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:21:51.898427 18784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567: Generated new TSK 0
I20260810 06:21:51.899075 18784 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
I20260810 06:21:52.575187 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.204.212:35787/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34795
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:41771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:21:52.721181 18838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:21:52.721405 18838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:21:52.721462 18838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:21:52.725425 18838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:21:52.725477 18838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:21:52.725550 18838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:21:52.731592 18838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:41771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.204.212:35787/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:21:52.732869 18838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:21:52.734135 18838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:21:52.745425 18855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:52.745425 18851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:52.745661 18852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:52.745775 18838 server_base.cc:1034] running on GCE node
I20260810 06:21:52.746306 18838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:21:52.747120 18838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:21:52.748359 18838 hybrid_clock.cc:648] HybridClock initialized: now 1786342912748335 us; error 37 us; skew 500 ppm
Aug 10 06:21:52 dist-test-slave-q9h9 krb5kdc[17947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342912, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:21:52.752103 18838 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:21:52.753101 18838 webserver.cc:460] Webserver started at http://127.31.204.193:39561/ using document root <none> and password file <none>
I20260810 06:21:52.753753 18838 fs_manager.cc:359] Metadata directory not provided
I20260810 06:21:52.753832 18838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:21:52.754079 18838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 06:21:52 dist-test-slave-q9h9 krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342912, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
I20260810 06:21:52.774150 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "41a8f392961a403c845f3852a962b9ce"
format_stamp: "Formatted at 2026-08-10 06:21:52 on dist-test-slave-q9h9"
server_key: "94e7ff722a810b92768d6031ac138a92"
server_key_iv: "c4ec8da175abc4f732f5851c8134a0ee"
server_key_version: "kuduclusterkey@0"
I20260810 06:21:52.775017 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "41a8f392961a403c845f3852a962b9ce"
format_stamp: "Formatted at 2026-08-10 06:21:52 on dist-test-slave-q9h9"
server_key: "94e7ff722a810b92768d6031ac138a92"
server_key_iv: "c4ec8da175abc4f732f5851c8134a0ee"
server_key_version: "kuduclusterkey@0"
I20260810 06:21:52.781538 18838 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260810 06:21:52.798599 18867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:52.800246 18838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 06:21:52.800379 18838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "41a8f392961a403c845f3852a962b9ce"
format_stamp: "Formatted at 2026-08-10 06:21:52 on dist-test-slave-q9h9"
server_key: "94e7ff722a810b92768d6031ac138a92"
server_key_iv: "c4ec8da175abc4f732f5851c8134a0ee"
server_key_version: "kuduclusterkey@0"
I20260810 06:21:52.800503 18838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:21:52.832885 18838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:21:52.836088 18838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:21:52.836344 18838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:21:52.837253 18838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:21:52.838881 18838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:21:52.839022 18838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:52.839154 18838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:21:52.839249 18838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:52.855623 18838 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:35229
I20260810 06:21:52.855836 19053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:35229 every 8 connection(s)
I20260810 06:21:52.856628 18838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:21:52.863806 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 18838
I20260810 06:21:52.863936 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:21:52.864245 32563 external_mini_cluster.cc:1428] Setting key becdd55800ab21b85ca74a1b8639a0b8
Aug 10 06:21:52 dist-test-slave-q9h9 krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342912, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:21:52.875783 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:52.859125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34795 (local address 127.31.204.193:50379)
0810 06:21:52.859706 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:52.859775 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:52.860843 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:52.861414 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:52.861423 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:52.861785 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:52.862484 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:52.862504 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:52.863610 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:52.863614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:52.864985 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:52.864996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:52.865787 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:52.865791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:52.865986 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:52.867563 (+ 1577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:52.867585 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:52.869974 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:21:52.872274 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:21:52.872280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:21:52.872295 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:21:52.872642 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:21:52.872967 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:21:52.872970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:21:52.872973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:21:52.873161 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:21:52.873649 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:21:52.873676 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:21:52.874060 (+ 384us) client_negotiation.cc:769] Sending connection context
0810 06:21:52.874358 (+ 298us) client_negotiation.cc:240] Negotiation successful
0810 06:21:52.874707 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":137,"threads_started":1}
I20260810 06:21:52.876506 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:52.858627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:50379 (local address 127.31.204.254:34795)
0810 06:21:52.858902 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:52.858910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:52.859881 (+ 971us) server_negotiation.cc:407] Connection header received
0810 06:21:52.861081 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:52.861084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:52.861136 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:52.861223 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:52.862686 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:52.863505 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:52.865133 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:52.865498 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:52.865577 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:52.870196 (+ 4619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:21:52.870200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:21:52.870203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:21:52.870243 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:21:52.872135 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:52.872764 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:52.872768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:52.872770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:52.872867 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:52.873274 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:52.873278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:52.873280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:52.873500 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:21:52.873589 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:52.876209 (+ 2620us) server_negotiation.cc:299] Negotiation successful
0810 06:21:52.876339 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":100,"threads_started":1}
I20260810 06:21:52.878264 19054 heartbeater.cc:347] Connected to a master server at 127.31.204.254:34795
I20260810 06:21:52.878504 19054 heartbeater.cc:464] Registering TS with master...
I20260810 06:21:52.879061 19054 heartbeater.cc:511] Master 127.31.204.254:34795 requested a full tablet report, sending...
I20260810 06:21:52.880937 18710 ts_manager.cc:194] Registered new tserver with Master: 41a8f392961a403c845f3852a962b9ce (127.31.204.193:35229)
I20260810 06:21:52.882498 18710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:50379
WARNING: no policy specified for kudu/127.31.204.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
I20260810 06:21:52.902081 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.204.212:35787/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34795
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:41771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 06:21:52 dist-test-slave-q9h9 krb5kdc[17947](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786342912, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 06:21:53.048821 19066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:21:53.049165 19066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:21:53.049228 19066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:21:53.055127 19066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:21:53.055214 19066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:21:53.055299 19066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:21:53.062932 19066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:41771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.204.212:35787/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:21:53.064342 19066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:21:53.065753 19066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:21:53.077538 19077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:53.077638 19075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:53.077534 19073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:53.078788 19066 server_base.cc:1034] running on GCE node
I20260810 06:21:53.079241 19066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:21:53.080032 19066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:21:53.081202 19066 hybrid_clock.cc:648] HybridClock initialized: now 1786342913081165 us; error 54 us; skew 500 ppm
Aug 10 06:21:53 dist-test-slave-q9h9 krb5kdc[17947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342913, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:21:53.084758 19066 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:21:53.085896 19066 webserver.cc:460] Webserver started at http://127.31.204.194:44303/ using document root <none> and password file <none>
I20260810 06:21:53.086696 19066 fs_manager.cc:359] Metadata directory not provided
I20260810 06:21:53.086797 19066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:21:53.087126 19066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 06:21:53 dist-test-slave-q9h9 krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342913, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
I20260810 06:21:53.102897 19066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "13bdbbe1b8134564914f8062b436ebcd"
format_stamp: "Formatted at 2026-08-10 06:21:53 on dist-test-slave-q9h9"
server_key: "aaa22406e528916e06bc314ae4cc1a56"
server_key_iv: "1c43870b7f7373eda4aac43a185bfe3e"
server_key_version: "kuduclusterkey@0"
I20260810 06:21:53.103789 19066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "13bdbbe1b8134564914f8062b436ebcd"
format_stamp: "Formatted at 2026-08-10 06:21:53 on dist-test-slave-q9h9"
server_key: "aaa22406e528916e06bc314ae4cc1a56"
server_key_iv: "1c43870b7f7373eda4aac43a185bfe3e"
server_key_version: "kuduclusterkey@0"
I20260810 06:21:53.108464 19066 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 06:21:53.141175 19089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:53.142933 19066 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260810 06:21:53.143090 19066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "13bdbbe1b8134564914f8062b436ebcd"
format_stamp: "Formatted at 2026-08-10 06:21:53 on dist-test-slave-q9h9"
server_key: "aaa22406e528916e06bc314ae4cc1a56"
server_key_iv: "1c43870b7f7373eda4aac43a185bfe3e"
server_key_version: "kuduclusterkey@0"
I20260810 06:21:53.143234 19066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:21:53.158773 19066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:21:53.162593 19066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:21:53.162842 19066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:21:53.163559 19066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:21:53.164708 19066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:21:53.164808 19066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:53.164901 19066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:21:53.164956 19066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:53.176419 19066 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:34923
I20260810 06:21:53.177646 19066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:21:53.180572 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 19066
I20260810 06:21:53.180711 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:21:53.181047 32563 external_mini_cluster.cc:1428] Setting key 80880e2ccf02bb442c961b60cee6307c
I20260810 06:21:53.186818 19257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:34923 every 8 connection(s)
WARNING: no policy specified for kudu/127.31.204.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.195@KRBTEST.COM" created.
Aug 10 06:21:53 dist-test-slave-q9h9 krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342913, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:21:53.210278 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:53.193615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:39803 (local address 127.31.204.254:34795)
0810 06:21:53.193758 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:53.193762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:53.194699 (+ 937us) server_negotiation.cc:407] Connection header received
0810 06:21:53.196107 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:53.196112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:53.196168 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:53.196229 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:53.200037 (+ 3808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.200883 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.202611 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.202976 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.203051 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:53.206088 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:21:53.206093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:21:53.206095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:21:53.206125 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:21:53.207836 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:53.208456 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:53.208460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:53.208462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:53.208548 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:53.208933 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:53.208937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:53.208938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:53.209157 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:21:53.209257 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:53.209906 (+ 649us) server_negotiation.cc:299] Negotiation successful
0810 06:21:53.210063 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:21:53.212085 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:53.192425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34795 (local address 127.31.204.194:39803)
0810 06:21:53.194543 (+ 2118us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:53.194598 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:53.195714 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:53.196632 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:53.196640 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:53.197084 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:53.199861 (+ 2777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:53.199879 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.201047 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.201050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:53.202469 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:53.202481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.203264 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.203267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:53.203465 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:53.204131 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:53.204152 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:53.205862 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:21:53.207976 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:21:53.207982 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:21:53.207998 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:21:53.208340 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:21:53.208647 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:21:53.208650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:21:53.208652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:21:53.208825 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:21:53.209327 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:21:53.209334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:21:53.209633 (+ 299us) client_negotiation.cc:769] Sending connection context
0810 06:21:53.210733 (+ 1100us) client_negotiation.cc:240] Negotiation successful
0810 06:21:53.211069 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1884,"thread_start_us":170,"threads_started":1}
I20260810 06:21:53.215704 19258 heartbeater.cc:347] Connected to a master server at 127.31.204.254:34795
I20260810 06:21:53.215948 19258 heartbeater.cc:464] Registering TS with master...
I20260810 06:21:53.216531 19258 heartbeater.cc:511] Master 127.31.204.254:34795 requested a full tablet report, sending...
I20260810 06:21:53.217861 18710 ts_manager.cc:194] Registered new tserver with Master: 13bdbbe1b8134564914f8062b436ebcd (127.31.204.194:34923)
I20260810 06:21:53.218679 18710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:39803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
I20260810 06:21:53.233855 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.204.212:35787/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34795
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:41771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:21:53.391086 19266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:21:53.391490 19266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:21:53.391603 19266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:21:53.397434 19266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:21:53.397565 19266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:21:53.397698 19266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:21:53.405197 19266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:41771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.204.212:35787/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:21:53.406760 19266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
Aug 10 06:21:53 dist-test-slave-q9h9 krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342912, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
I20260810 06:21:53.408078 19266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:21:53.420935 19279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:21:53.421494 19277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:53.422672 19266 server_base.cc:1034] running on GCE node
W20260810 06:21:53.438674 19281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:21:53.439349 19266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:21:53.440135 19266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:21:53.441672 19266 hybrid_clock.cc:648] HybridClock initialized: now 1786342913441588 us; error 102 us; skew 500 ppm
Aug 10 06:21:53 dist-test-slave-q9h9 krb5kdc[17947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342913, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:21:53.445619 19266 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:21:53.446748 19266 webserver.cc:460] Webserver started at http://127.31.204.195:42769/ using document root <none> and password file <none>
I20260810 06:21:53.447520 19266 fs_manager.cc:359] Metadata directory not provided
I20260810 06:21:53.447609 19266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:21:53.447834 19266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 06:21:53 dist-test-slave-q9h9 krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342913, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
I20260810 06:21:53.469311 19266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "321305bbe7974e53bda094c1f8b9f9e6"
format_stamp: "Formatted at 2026-08-10 06:21:53 on dist-test-slave-q9h9"
server_key: "8117b0cf519fcd6af6d539a86c669dab"
server_key_iv: "044ba1f6c484ca5744b8e6748f24fe7a"
server_key_version: "kuduclusterkey@0"
I20260810 06:21:53.470356 19266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "321305bbe7974e53bda094c1f8b9f9e6"
format_stamp: "Formatted at 2026-08-10 06:21:53 on dist-test-slave-q9h9"
server_key: "8117b0cf519fcd6af6d539a86c669dab"
server_key_iv: "044ba1f6c484ca5744b8e6748f24fe7a"
server_key_version: "kuduclusterkey@0"
I20260810 06:21:53.483069 19266 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.000s
I20260810 06:21:53.505342 19292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:53.507088 19266 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260810 06:21:53.507249 19266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "321305bbe7974e53bda094c1f8b9f9e6"
format_stamp: "Formatted at 2026-08-10 06:21:53 on dist-test-slave-q9h9"
server_key: "8117b0cf519fcd6af6d539a86c669dab"
server_key_iv: "044ba1f6c484ca5744b8e6748f24fe7a"
server_key_version: "kuduclusterkey@0"
I20260810 06:21:53.507403 19266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:21:53.524574 19266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:21:53.528582 19266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:21:53.528805 19266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:21:53.529531 19266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:21:53.530807 19266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:21:53.530885 19266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:53.530959 19266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:21:53.530997 19266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:21:53.554225 19266 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:40959
I20260810 06:21:53.554256 19475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:40959 every 8 connection(s)
I20260810 06:21:53.555606 19266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:21:53.561872 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 19266
I20260810 06:21:53.562004 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:21:53.562337 32563 external_mini_cluster.cc:1428] Setting key ab3d9ae57bb5e740dcff1382464cb781
Aug 10 06:21:53 dist-test-slave-q9h9 krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342913, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:21:53.573056 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:53.558133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:55179 (local address 127.31.204.254:34795)
0810 06:21:53.558265 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:53.558269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:53.559560 (+ 1291us) server_negotiation.cc:407] Connection header received
0810 06:21:53.561022 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:53.561026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:53.561094 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:53.561195 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:53.562785 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.563659 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.565652 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.566054 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.566171 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:53.569012 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:21:53.569016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:21:53.569019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:21:53.569049 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:21:53.570724 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:53.571335 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:53.571338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:53.571340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:53.571420 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:53.571812 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:53.571814 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:53.571816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:53.572022 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:21:53.572134 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:53.572780 (+ 646us) server_negotiation.cc:299] Negotiation successful
0810 06:21:53.572898 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 06:21:53.574160 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:53.558693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34795 (local address 127.31.204.195:55179)
0810 06:21:53.559395 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:53.559452 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:53.560769 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:53.561380 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:53.561389 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:53.561819 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:21:53.562555 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:53.562575 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.563988 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.563991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:53.565476 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:53.565487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.566213 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.566217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:53.566414 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:53.567053 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:21:53.567074 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:21:53.568839 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:21:53.570849 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:21:53.570855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:21:53.570873 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:21:53.571233 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:21:53.571503 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:21:53.571506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:21:53.571509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:21:53.571716 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:21:53.572153 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:21:53.572159 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:21:53.572498 (+ 339us) client_negotiation.cc:769] Sending connection context
0810 06:21:53.572730 (+ 232us) client_negotiation.cc:240] Negotiation successful
0810 06:21:53.573060 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":285,"threads_started":1}
I20260810 06:21:53.575312 19476 heartbeater.cc:347] Connected to a master server at 127.31.204.254:34795
I20260810 06:21:53.575567 19476 heartbeater.cc:464] Registering TS with master...
I20260810 06:21:53.576133 19476 heartbeater.cc:511] Master 127.31.204.254:34795 requested a full tablet report, sending...
I20260810 06:21:53.577329 18710 ts_manager.cc:194] Registered new tserver with Master: 321305bbe7974e53bda094c1f8b9f9e6 (127.31.204.195:40959)
I20260810 06:21:53.578094 18710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:55179
I20260810 06:21:53.579068 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:21:53.591704 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:53.582160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49766 (local address 127.31.204.254:34795)
0810 06:21:53.582290 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:53.582294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:53.582920 (+ 626us) server_negotiation.cc:407] Connection header received
0810 06:21:53.583046 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:53.583049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:53.583103 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:53.583183 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:21:53.584070 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.584855 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.586055 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.586398 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.586487 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:53.588237 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:21:53.588241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:21:53.588244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:21:53.588270 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:21:53.589907 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:53.590428 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:53.590431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:53.590433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:53.590516 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:21:53.590820 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:21:53.590823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:21:53.590824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:21:53.591023 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:21:53.591157 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:53.591432 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 06:21:53.591545 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:21:53.595026 18710 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:21:53.599128 18710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:21:53.619663 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:53.609365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:34923 (local address 127.0.0.1:58570)
0810 06:21:53.611845 (+ 2480us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:53.611906 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:53.612093 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:53.612898 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:53.612905 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:53.612933 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:21:53.613947 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:53.613955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.615467 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.615471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:53.617738 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:53.617750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.619111 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.619114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:53.619232 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:53.619271 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 06:21:53.619381 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 06:21:53.619503 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2261,"thread_start_us":99,"threads_started":1}
I20260810 06:21:53.621052 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:53.609613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58570 (local address 127.31.204.194:34923)
0810 06:21:53.610221 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:53.610234 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:53.612262 (+ 2028us) server_negotiation.cc:407] Connection header received
0810 06:21:53.612434 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:53.612440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:53.612696 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:53.612951 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:21:53.614124 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.615322 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.617910 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.618780 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.618890 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:53.619038 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:53.620009 (+ 971us) server_negotiation.cc:299] Negotiation successful
0810 06:21:53.620135 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":248,"threads_started":1}
I20260810 06:21:53.624394 19144 tablet_service.cc:1467] Processing CreateTablet for tablet 26dcb57f37e8471dac0d32ee1c41ced4 (DEFAULT_TABLE table=test-table [id=29e8f50e26a244b19523085b656f35ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:21:53.626376 19144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26dcb57f37e8471dac0d32ee1c41ced4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:21:53.630246 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:53.611668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:40959 (local address 127.0.0.1:46232)
0810 06:21:53.618298 (+ 6630us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:53.618320 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:53.618447 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:53.619555 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:53.619558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:53.619577 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:21:53.619679 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:53.619684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.622895 (+ 3211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.622899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:53.624107 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:53.624120 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.629671 (+ 5551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.629675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:53.629835 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:53.629899 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 06:21:53.629991 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 06:21:53.630095 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6531,"thread_start_us":97,"threads_started":1}
I20260810 06:21:53.631278 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:53.612118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46232 (local address 127.31.204.195:40959)
0810 06:21:53.619080 (+ 6962us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:53.619094 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:53.619120 (+ 26us) server_negotiation.cc:407] Connection header received
0810 06:21:53.619196 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:53.619202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:53.619402 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:53.620317 (+ 915us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:21:53.620455 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.622756 (+ 2301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.624289 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.625281 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.630564 (+ 5283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:53.630831 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:53.630916 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 06:21:53.631050 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6524,"thread_start_us":162,"threads_started":1}
I20260810 06:21:53.635099 19365 tablet_service.cc:1467] Processing CreateTablet for tablet 26dcb57f37e8471dac0d32ee1c41ced4 (DEFAULT_TABLE table=test-table [id=29e8f50e26a244b19523085b656f35ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:21:53.637192 19365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26dcb57f37e8471dac0d32ee1c41ced4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:21:53.638320 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:53.610720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:35229 (local address 127.0.0.1:45258)
0810 06:21:53.613262 (+ 2542us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:53.613283 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:53.613369 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:53.618792 (+ 5423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:53.618795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:53.618818 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:21:53.618939 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:53.618945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.635623 (+ 16678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.635626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:53.636722 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:53.636733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.637907 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.637910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:53.638009 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:53.638052 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 06:21:53.638119 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 06:21:53.638186 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2458,"thread_start_us":249,"threads_started":1}
I20260810 06:21:53.638960 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:53.610860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45258 (local address 127.31.204.193:35229)
0810 06:21:53.618266 (+ 7406us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:53.618274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:53.618301 (+ 27us) server_negotiation.cc:407] Connection header received
0810 06:21:53.618385 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:53.618391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:53.618649 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:53.618839 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:21:53.633064 (+ 14225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.634269 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.636865 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.637618 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.637729 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:53.637847 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:53.638720 (+ 873us) server_negotiation.cc:299] Negotiation successful
0810 06:21:53.638823 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7262,"thread_start_us":100,"threads_started":1}
I20260810 06:21:53.642194 19504 tablet_bootstrap.cc:492] T 26dcb57f37e8471dac0d32ee1c41ced4 P 13bdbbe1b8134564914f8062b436ebcd: Bootstrap starting.
I20260810 06:21:53.642201 18937 tablet_service.cc:1467] Processing CreateTablet for tablet 26dcb57f37e8471dac0d32ee1c41ced4 (DEFAULT_TABLE table=test-table [id=29e8f50e26a244b19523085b656f35ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:21:53.643334 18937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26dcb57f37e8471dac0d32ee1c41ced4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:21:53.645332 19504 tablet_bootstrap.cc:654] T 26dcb57f37e8471dac0d32ee1c41ced4 P 13bdbbe1b8134564914f8062b436ebcd: Neither blocks nor log segments found. Creating new log.
I20260810 06:21:53.646252 19504 log.cc:824] T 26dcb57f37e8471dac0d32ee1c41ced4 P 13bdbbe1b8134564914f8062b436ebcd: Log is configured to *not* fsync() on all Append() calls
I20260810 06:21:53.652616 19507 tablet_bootstrap.cc:492] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6: Bootstrap starting.
I20260810 06:21:53.653908 19504 tablet_bootstrap.cc:492] T 26dcb57f37e8471dac0d32ee1c41ced4 P 13bdbbe1b8134564914f8062b436ebcd: No bootstrap required, opened a new log
I20260810 06:21:53.654126 19504 ts_tablet_manager.cc:1397] T 26dcb57f37e8471dac0d32ee1c41ced4 P 13bdbbe1b8134564914f8062b436ebcd: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260810 06:21:53.655598 19507 tablet_bootstrap.cc:654] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6: Neither blocks nor log segments found. Creating new log.
I20260810 06:21:53.656687 19507 log.cc:824] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6: Log is configured to *not* fsync() on all Append() calls
I20260810 06:21:53.657681 19506 tablet_bootstrap.cc:492] T 26dcb57f37e8471dac0d32ee1c41ced4 P 41a8f392961a403c845f3852a962b9ce: Bootstrap starting.
I20260810 06:21:53.659041 19504 raft_consensus.cc:348] T 26dcb57f37e8471dac0d32ee1c41ced4 P 13bdbbe1b8134564914f8062b436ebcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13bdbbe1b8134564914f8062b436ebcd" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34923 } } peers { permanent_uuid: "41a8f392961a403c845f3852a962b9ce" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 35229 } } peers { permanent_uuid: "321305bbe7974e53bda094c1f8b9f9e6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40959 } }
I20260810 06:21:53.659291 19504 raft_consensus.cc:374] T 26dcb57f37e8471dac0d32ee1c41ced4 P 13bdbbe1b8134564914f8062b436ebcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:21:53.659330 19504 raft_consensus.cc:729] T 26dcb57f37e8471dac0d32ee1c41ced4 P 13bdbbe1b8134564914f8062b436ebcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13bdbbe1b8134564914f8062b436ebcd, State: Initialized, Role: FOLLOWER
I20260810 06:21:53.659937 19506 tablet_bootstrap.cc:654] T 26dcb57f37e8471dac0d32ee1c41ced4 P 41a8f392961a403c845f3852a962b9ce: Neither blocks nor log segments found. Creating new log.
I20260810 06:21:53.659979 19504 consensus_queue.cc:260] T 26dcb57f37e8471dac0d32ee1c41ced4 P 13bdbbe1b8134564914f8062b436ebcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13bdbbe1b8134564914f8062b436ebcd" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34923 } } peers { permanent_uuid: "41a8f392961a403c845f3852a962b9ce" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 35229 } } peers { permanent_uuid: "321305bbe7974e53bda094c1f8b9f9e6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40959 } }
I20260810 06:21:53.660840 19506 log.cc:824] T 26dcb57f37e8471dac0d32ee1c41ced4 P 41a8f392961a403c845f3852a962b9ce: Log is configured to *not* fsync() on all Append() calls
I20260810 06:21:53.661072 19504 ts_tablet_manager.cc:1428] T 26dcb57f37e8471dac0d32ee1c41ced4 P 13bdbbe1b8134564914f8062b436ebcd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 06:21:53.661456 19258 heartbeater.cc:503] Master 127.31.204.254:34795 was elected leader, sending a full tablet report...
I20260810 06:21:53.663178 19507 tablet_bootstrap.cc:492] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6: No bootstrap required, opened a new log
I20260810 06:21:53.663360 19507 ts_tablet_manager.cc:1397] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260810 06:21:53.667222 19507 raft_consensus.cc:348] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13bdbbe1b8134564914f8062b436ebcd" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34923 } } peers { permanent_uuid: "41a8f392961a403c845f3852a962b9ce" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 35229 } } peers { permanent_uuid: "321305bbe7974e53bda094c1f8b9f9e6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40959 } }
I20260810 06:21:53.668174 19507 raft_consensus.cc:374] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:21:53.668282 19507 raft_consensus.cc:729] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 321305bbe7974e53bda094c1f8b9f9e6, State: Initialized, Role: FOLLOWER
I20260810 06:21:53.668784 19507 consensus_queue.cc:260] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13bdbbe1b8134564914f8062b436ebcd" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34923 } } peers { permanent_uuid: "41a8f392961a403c845f3852a962b9ce" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 35229 } } peers { permanent_uuid: "321305bbe7974e53bda094c1f8b9f9e6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40959 } }
I20260810 06:21:53.669045 19506 tablet_bootstrap.cc:492] T 26dcb57f37e8471dac0d32ee1c41ced4 P 41a8f392961a403c845f3852a962b9ce: No bootstrap required, opened a new log
I20260810 06:21:53.669308 19506 ts_tablet_manager.cc:1397] T 26dcb57f37e8471dac0d32ee1c41ced4 P 41a8f392961a403c845f3852a962b9ce: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260810 06:21:53.669880 19507 ts_tablet_manager.cc:1428] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 06:21:53.670267 19476 heartbeater.cc:503] Master 127.31.204.254:34795 was elected leader, sending a full tablet report...
I20260810 06:21:53.674937 19506 raft_consensus.cc:348] T 26dcb57f37e8471dac0d32ee1c41ced4 P 41a8f392961a403c845f3852a962b9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13bdbbe1b8134564914f8062b436ebcd" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34923 } } peers { permanent_uuid: "41a8f392961a403c845f3852a962b9ce" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 35229 } } peers { permanent_uuid: "321305bbe7974e53bda094c1f8b9f9e6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40959 } }
I20260810 06:21:53.675279 19506 raft_consensus.cc:374] T 26dcb57f37e8471dac0d32ee1c41ced4 P 41a8f392961a403c845f3852a962b9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:21:53.675386 19506 raft_consensus.cc:729] T 26dcb57f37e8471dac0d32ee1c41ced4 P 41a8f392961a403c845f3852a962b9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a8f392961a403c845f3852a962b9ce, State: Initialized, Role: FOLLOWER
I20260810 06:21:53.675865 19506 consensus_queue.cc:260] T 26dcb57f37e8471dac0d32ee1c41ced4 P 41a8f392961a403c845f3852a962b9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13bdbbe1b8134564914f8062b436ebcd" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34923 } } peers { permanent_uuid: "41a8f392961a403c845f3852a962b9ce" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 35229 } } peers { permanent_uuid: "321305bbe7974e53bda094c1f8b9f9e6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40959 } }
I20260810 06:21:53.677114 19506 ts_tablet_manager.cc:1428] T 26dcb57f37e8471dac0d32ee1c41ced4 P 41a8f392961a403c845f3852a962b9ce: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260810 06:21:53.677491 19054 heartbeater.cc:503] Master 127.31.204.254:34795 was elected leader, sending a full tablet report...
W20260810 06:21:53.695348 19261 tablet.cc:2369] T 26dcb57f37e8471dac0d32ee1c41ced4 P 13bdbbe1b8134564914f8062b436ebcd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 06:21:53 dist-test-slave-q9h9 krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342912, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
W20260810 06:21:53.816663 19477 tablet.cc:2369] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:21:53.859150 19055 tablet.cc:2369] T 26dcb57f37e8471dac0d32ee1c41ced4 P 41a8f392961a403c845f3852a962b9ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:21:53.867426 19516 raft_consensus.cc:482] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:21:53.867640 19516 raft_consensus.cc:504] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13bdbbe1b8134564914f8062b436ebcd" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34923 } } peers { permanent_uuid: "41a8f392961a403c845f3852a962b9ce" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 35229 } } peers { permanent_uuid: "321305bbe7974e53bda094c1f8b9f9e6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40959 } }
I20260810 06:21:53.869026 19516 leader_election.cc:288] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13bdbbe1b8134564914f8062b436ebcd (127.31.204.194:34923), 41a8f392961a403c845f3852a962b9ce (127.31.204.193:35229)
I20260810 06:21:53.875468 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:53.869500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:34923 (local address 127.31.204.195:48401)
0810 06:21:53.869902 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:53.869919 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:53.870045 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:53.871560 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:53.871564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:53.871587 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:21:53.871706 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:53.871714 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.872770 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.872773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:53.873988 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:53.874000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.875044 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.875048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:53.875167 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:53.875192 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 06:21:53.875265 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 06:21:53.875334 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310}
I20260810 06:21:53.876626 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:53.870233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:48401 (local address 127.31.204.194:34923)
0810 06:21:53.870362 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:53.870367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:53.870380 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:21:53.870431 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:53.870434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:53.870479 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:53.870547 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:21:53.871850 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.872634 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.874185 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.874783 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.874922 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:53.874979 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:53.876430 (+ 1451us) server_negotiation.cc:299] Negotiation successful
0810 06:21:53.876508 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 06:21:53.877296 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26dcb57f37e8471dac0d32ee1c41ced4" candidate_uuid: "321305bbe7974e53bda094c1f8b9f9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13bdbbe1b8134564914f8062b436ebcd" is_pre_election: true
I20260810 06:21:53.877650 19195 raft_consensus.cc:2454] T 26dcb57f37e8471dac0d32ee1c41ced4 P 13bdbbe1b8134564914f8062b436ebcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 321305bbe7974e53bda094c1f8b9f9e6 in term 0.
I20260810 06:21:53.878728 19293 leader_election.cc:302] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13bdbbe1b8134564914f8062b436ebcd, 321305bbe7974e53bda094c1f8b9f9e6; no voters:
I20260810 06:21:53.879144 19516 raft_consensus.cc:2781] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:21:53.879235 19516 raft_consensus.cc:482] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:21:53.879280 19516 raft_consensus.cc:3037] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:21:53.880721 19516 raft_consensus.cc:504] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13bdbbe1b8134564914f8062b436ebcd" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34923 } } peers { permanent_uuid: "41a8f392961a403c845f3852a962b9ce" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 35229 } } peers { permanent_uuid: "321305bbe7974e53bda094c1f8b9f9e6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40959 } }
I20260810 06:21:53.881362 19516 leader_election.cc:288] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [CANDIDATE]: Term 1 election: Requested vote from peers 13bdbbe1b8134564914f8062b436ebcd (127.31.204.194:34923), 41a8f392961a403c845f3852a962b9ce (127.31.204.193:35229)
I20260810 06:21:53.881788 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26dcb57f37e8471dac0d32ee1c41ced4" candidate_uuid: "321305bbe7974e53bda094c1f8b9f9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13bdbbe1b8134564914f8062b436ebcd"
I20260810 06:21:53.881908 19195 raft_consensus.cc:3037] T 26dcb57f37e8471dac0d32ee1c41ced4 P 13bdbbe1b8134564914f8062b436ebcd [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:21:53.883208 19195 raft_consensus.cc:2454] T 26dcb57f37e8471dac0d32ee1c41ced4 P 13bdbbe1b8134564914f8062b436ebcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 321305bbe7974e53bda094c1f8b9f9e6 in term 1.
I20260810 06:21:53.883608 19293 leader_election.cc:302] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13bdbbe1b8134564914f8062b436ebcd, 321305bbe7974e53bda094c1f8b9f9e6; no voters:
I20260810 06:21:53.885934 19516 raft_consensus.cc:2781] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:21:53.886354 19516 raft_consensus.cc:686] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [term 1 LEADER]: Becoming Leader. State: Replica: 321305bbe7974e53bda094c1f8b9f9e6, State: Running, Role: LEADER
I20260810 06:21:53.887018 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:53.871383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:34115 (local address 127.31.204.193:35229)
0810 06:21:53.871497 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:53.871501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:53.875874 (+ 4373us) server_negotiation.cc:407] Connection header received
0810 06:21:53.875929 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:53.875932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:53.875979 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:53.876047 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:21:53.877368 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.878280 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.884915 (+ 6635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.885525 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.885647 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:53.885702 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:53.886809 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0810 06:21:53.886893 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:21:53.887670 19516 consensus_queue.cc:237] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13bdbbe1b8134564914f8062b436ebcd" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34923 } } peers { permanent_uuid: "41a8f392961a403c845f3852a962b9ce" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 35229 } } peers { permanent_uuid: "321305bbe7974e53bda094c1f8b9f9e6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40959 } }
I20260810 06:21:53.888053 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:53.870829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:35229 (local address 127.31.204.195:34115)
0810 06:21:53.875618 (+ 4789us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:53.875633 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:21:53.875712 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:21:53.877142 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:21:53.877145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:21:53.877165 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:21:53.877238 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:53.877244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.883704 (+ 6460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.883707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:53.884761 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:21:53.884774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.886504 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.886508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:21:53.886597 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:53.886624 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 06:21:53.886700 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 06:21:53.887893 (+ 1193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4735,"thread_start_us":279,"threads_started":1}
I20260810 06:21:53.889315 18979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26dcb57f37e8471dac0d32ee1c41ced4" candidate_uuid: "321305bbe7974e53bda094c1f8b9f9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a8f392961a403c845f3852a962b9ce" is_pre_election: true
I20260810 06:21:53.889640 18979 raft_consensus.cc:2454] T 26dcb57f37e8471dac0d32ee1c41ced4 P 41a8f392961a403c845f3852a962b9ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 321305bbe7974e53bda094c1f8b9f9e6 in term 0.
I20260810 06:21:53.890002 18978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26dcb57f37e8471dac0d32ee1c41ced4" candidate_uuid: "321305bbe7974e53bda094c1f8b9f9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a8f392961a403c845f3852a962b9ce"
I20260810 06:21:53.890156 18978 raft_consensus.cc:3037] T 26dcb57f37e8471dac0d32ee1c41ced4 P 41a8f392961a403c845f3852a962b9ce [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:21:53.891479 18978 raft_consensus.cc:2454] T 26dcb57f37e8471dac0d32ee1c41ced4 P 41a8f392961a403c845f3852a962b9ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 321305bbe7974e53bda094c1f8b9f9e6 in term 1.
I20260810 06:21:53.893188 18710 catalog_manager.cc:5495] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 321305bbe7974e53bda094c1f8b9f9e6 (127.31.204.195). New cstate: current_term: 1 leader_uuid: "321305bbe7974e53bda094c1f8b9f9e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13bdbbe1b8134564914f8062b436ebcd" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41a8f392961a403c845f3852a962b9ce" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 35229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "321305bbe7974e53bda094c1f8b9f9e6" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40959 } health_report { overall_health: HEALTHY } } }
I20260810 06:21:53.928201 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:53.922858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46236 (local address 127.31.204.195:40959)
0810 06:21:53.922986 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 06:21:53.922991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:21:53.923010 (+ 19us) server_negotiation.cc:407] Connection header received
0810 06:21:53.923263 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:21:53.923267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:21:53.923341 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:21:53.923447 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:21:53.923873 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.924661 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.925939 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:21:53.926307 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:21:53.926410 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:21:53.926849 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:21:53.927423 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:21:53.927535 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 06:21:53.927952 (+ 417us) server_negotiation.cc:299] Negotiation successful
0810 06:21:53.928025 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 06:21:53.935599 18978 raft_consensus.cc:1261] T 26dcb57f37e8471dac0d32ee1c41ced4 P 41a8f392961a403c845f3852a962b9ce [term 1 FOLLOWER]: Refusing update from remote peer 321305bbe7974e53bda094c1f8b9f9e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:21:53.935614 19195 raft_consensus.cc:1261] T 26dcb57f37e8471dac0d32ee1c41ced4 P 13bdbbe1b8134564914f8062b436ebcd [term 1 FOLLOWER]: Refusing update from remote peer 321305bbe7974e53bda094c1f8b9f9e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:21:53.936408 19527 consensus_queue.cc:1035] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13bdbbe1b8134564914f8062b436ebcd" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:21:53.936671 19516 consensus_queue.cc:1035] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41a8f392961a403c845f3852a962b9ce" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 35229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:21:53.947585 19532 mvcc.cc:205] Tried to move back new op lower bound from 7316860575471878144 to 7316860575292420096. Current Snapshot: MvccSnapshot[applied={T|T < 7316860575471878144}]
I20260810 06:21:53.957415 18710 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:21:53.957588 18710 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:21:53.960384 18710 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e240096bf3954611930ba59088672567: Sending DeleteTablet for 3 replicas of tablet 26dcb57f37e8471dac0d32ee1c41ced4
I20260810 06:21:53.961089 19144 tablet_service.cc:1514] Processing DeleteTablet for tablet 26dcb57f37e8471dac0d32ee1c41ced4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:21:53 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:58570
I20260810 06:21:53.962085 18937 tablet_service.cc:1514] Processing DeleteTablet for tablet 26dcb57f37e8471dac0d32ee1c41ced4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:21:53 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:45258
I20260810 06:21:53.963176 19365 tablet_service.cc:1514] Processing DeleteTablet for tablet 26dcb57f37e8471dac0d32ee1c41ced4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:21:53 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:46232
I20260810 06:21:53.964672 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 18838
I20260810 06:21:53.981781 19552 tablet_replica.cc:330] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6: stopping tablet replica
I20260810 06:21:53.982156 19552 raft_consensus.cc:2229] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:21:53.982527 19552 raft_consensus.cc:2258] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:21:53.987572 19552 ts_tablet_manager.cc:1905] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:21:53.987869 19548 tablet_replica.cc:330] T 26dcb57f37e8471dac0d32ee1c41ced4 P 13bdbbe1b8134564914f8062b436ebcd: stopping tablet replica
I20260810 06:21:53.988281 19548 raft_consensus.cc:2229] T 26dcb57f37e8471dac0d32ee1c41ced4 P 13bdbbe1b8134564914f8062b436ebcd [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 06:21:53.988673 18687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.204.193:35229 (error 108)
I20260810 06:21:53.988739 19548 raft_consensus.cc:2258] T 26dcb57f37e8471dac0d32ee1c41ced4 P 13bdbbe1b8134564914f8062b436ebcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:21:53.989599 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 19066
I20260810 06:21:53.999630 19552 ts_tablet_manager.cc:1918] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:21:53.999784 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:53.999412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:35229 (local address 127.0.0.1:45266)
0810 06:21:53.999608 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:53.999687 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:35229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260810 06:21:53.999797 19552 log.cc:1196] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/26dcb57f37e8471dac0d32ee1c41ced4
W20260810 06:21:54.000022 18687 catalog_manager.cc:4584] TS 41a8f392961a403c845f3852a962b9ce (127.31.204.193:35229): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 26dcb57f37e8471dac0d32ee1c41ced4: Network error: Client connection negotiation failed: client connection to 127.31.204.193:35229: connect: Connection refused (error 111)
I20260810 06:21:54.000175 19552 ts_tablet_manager.cc:1939] T 26dcb57f37e8471dac0d32ee1c41ced4 P 321305bbe7974e53bda094c1f8b9f9e6: Deleting consensus metadata
I20260810 06:21:54.001641 18687 catalog_manager.cc:4841] TS 321305bbe7974e53bda094c1f8b9f9e6 (127.31.204.195:40959): tablet 26dcb57f37e8471dac0d32ee1c41ced4 (table test-table [id=29e8f50e26a244b19523085b656f35ed]) successfully deleted
I20260810 06:21:54.002880 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 19266
I20260810 06:21:54.003033 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:21:54.002770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:34923 (local address 127.0.0.1:58578)
0810 06:21:54.002927 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 06:21:54.002975 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.194:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260810 06:21:54.011535 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 18657
I20260810 06:21:54.033439 32563 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 06:21:54.377416 32563 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 06:21:54.377561 32563 mini_ranger.cc:66] Stopping Ranger...
I20260810 06:21:54.721495 32563 mini_ranger.cc:68] Stopped Ranger
2026-08-10 06:21:54.721 UTC [17994] LOG: received smart shutdown request
2026-08-10 06:21:54.725 UTC [17994] LOG: background worker "logical replication launcher" (PID 18009) exited with exit code 1
2026-08-10 06:21:54.725 UTC [18002] LOG: shutting down
2026-08-10 06:21:54.833 UTC [17994] LOG: database system is shut down
2026-08-10T06:21:54Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45389 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:21:54 dist-test-slave-q9h9 krb5kdc[19576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:21:54 dist-test-slave-q9h9 krb5kdc[19576](info): set up 2 sockets
Aug 10 06:21:54 dist-test-slave-q9h9 krb5kdc[19576](info): commencing operation
krb5kdc: starting...
W20260810 06:21:56.092207 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:21:56 dist-test-slave-q9h9 krb5kdc[19576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:21:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:21:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.204.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.204.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.204.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/rangeradmin_127.31.204.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.204.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/rangeradmin_127.31.204.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.204.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.204.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.204.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/rangerlookup_127.31.204.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.204.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/rangerlookup_127.31.204.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.204.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.204.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.204.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/HTTP_127.31.204.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.204.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/HTTP_127.31.204.212@KRBTEST.COM.keytab.
I20260810 06:21:56.240130 32563 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7DS5L1/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/postgres -l logfile start
2026-08-10 06:21:58.263 UTC [19624] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 06:21:58.263 UTC [19624] LOG: listening on IPv4 address "127.31.204.212", port 54373
2026-08-10 06:21:58.266 UTC [19624] LOG: listening on Unix socket "/tmp/.s.PGSQL.54373"
2026-08-10 06:21:58.285 UTC [19628] LOG: database system was shut down at 2026-08-10 06:21:57 UTC
2026-08-10 06:21:58.288 UTC [19624] LOG: database system is ready to accept connections
I20260810 06:21:59.357084 32563 mini_postgres.cc:96] Postgres bound to 54373
2026-08-10 06:21:59.364 UTC [19647] FATAL: database "slave" does not exist
127.31.204.212:54373 - accepting connections
I20260810 06:21:59.364727 32563 mini_ranger.cc:161] Starting Ranger...
I20260810 06:21:59.379434 32563 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 06:21:59.709007 32563 mini_ranger.cc:87] Created ranger Postgres database
I20260810 06:21:59.709095 32563 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ranger-admin
2026-08-10 06:21:59,973 [I] DB FLAVOR :POSTGRES
2026-08-10 06:21:59,974 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 06:21:59,974 [I] Checking connection..
2026-08-10 06:21:59,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 06:22:00,203 [I] Checking connection passed.
2026-08-10 06:22:00,204 [I] --------- Verifying version history table ---------
2026-08-10 06:22:00,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 06:22:00,441 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 06:22:00,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 06:22:00,699 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 06:22:00,699 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 06:22:00,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 06:22:00.943 UTC [19745] WARNING: there is no transaction in progress
2026-08-10 06:22:00,953 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 06:22:00,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 06:22:01,193 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 06:22:01,193 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 06:22:01,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 06:22:01,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 06:22:01,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q9h9.c.gcp-upstream.internal','N') ;"
2026-08-10 06:22:01,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 06:22:02,169 [I] Table x_portal_user does not exist in database ranger
2026-08-10 06:22:02,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 06:22:02,424 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 06:22:02,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 06:22:02,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 06:22:02,911 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 06:22:02,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 06:22:03.760 UTC [19946] WARNING: there is no transaction in progress
2026-08-10 06:22:03.773 UTC [19946] WARNING: there is no transaction in progress
2026-08-10 06:22:03.787 UTC [19946] WARNING: there is no transaction in progress
2026-08-10 06:22:03.799 UTC [19946] WARNING: there is no transaction in progress
2026-08-10 06:22:03.812 UTC [19946] WARNING: there is no transaction in progress
2026-08-10 06:22:03.827 UTC [19946] WARNING: there is no transaction in progress
2026-08-10 06:22:03.873 UTC [19946] WARNING: there is no transaction in progress
2026-08-10 06:22:03.879 UTC [19946] WARNING: there is no transaction in progress
2026-08-10 06:22:03.891 UTC [19946] WARNING: there is no transaction in progress
2026-08-10 06:22:03.899 UTC [19946] WARNING: there is no transaction in progress
2026-08-10 06:22:03.908 UTC [19946] WARNING: there is no transaction in progress
2026-08-10 06:22:03.919 UTC [19946] WARNING: there is no transaction in progress
2026-08-10 06:22:03.927 UTC [19946] WARNING: there is no transaction in progress
2026-08-10 06:22:03.932 UTC [19946] WARNING: there is no transaction in progress
2026-08-10 06:22:03.937 UTC [19946] WARNING: there is no transaction in progress
2026-08-10 06:22:03.942 UTC [19946] WARNING: there is no transaction in progress
2026-08-10 06:22:04,441 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 06:22:04,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 06:22:04,690 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 06:22:04,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 06:22:04,931 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 06:22:04,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 06:22:05,183 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 06:22:05,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 06:22:05,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 06:22:05,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q9h9.c.gcp-upstream.internal';"
2026-08-10 06:22:05,923 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 06:22:05,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 06:22:06,165 [I] DB_PATCHES have already been applied
I20260810 06:22:06.171018 32563 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ranger-admin:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7DS5L1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 06:22:06.171082 32563 mini_ranger.cc:193] Using host: 127.31.204.212
I20260810 06:22:06.175340 32563 mini_ranger.cc:227] Ranger admin URL: http://127.31.204.212:57543
Aug 10, 2026 6:22:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:22:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:22:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 6:22:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 6:22:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 6:22:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 6:22:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 06:22:07.405897 32563 mini_ranger.cc:160] Time spent starting Ranger: real 8.041s user 0.000s sys 0.011s
Aug 10, 2026 6:22:25 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 6:22:25 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 10, 2026 6:22:25 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 6:22:25 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 6:22:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 6:22:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 6:22:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 6:22:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 6:22:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 6:22:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 6:22:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 6:22:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 6:22:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 6:22:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 6:22:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 6:22:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 6:22:25 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 6:22:26 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 06:22:27.127265 32563 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.204.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.204.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.204.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/rangerkms_127.31.204.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.204.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/rangerkms_127.31.204.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.204.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.204.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.204.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/HTTP_127.31.204.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.204.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/HTTP_127.31.204.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 06:22:27 dist-test-slave-q9h9 krb5kdc[19576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342947, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 06:22:27.201733 32563 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 06:22:27.210578 32563 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 06:22:27.785220 32563 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 06:22:27.785311 32563 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ranger-kms
I20260810 06:22:27.785328 32563 mini_ranger_kms.cc:227] Using postgres at 127.31.204.212:54373
2026-08-10 06:22:27,948 [I] DB FLAVOR :POSTGRES
2026-08-10 06:22:27,949 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 06:22:27,949 [I] Checking connection
2026-08-10 06:22:27,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 06:22:28,173 [I] connection success
2026-08-10 06:22:28,173 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 06:22:28,173 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 06:22:28,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 06:22:28,399 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 06:22:28,399 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 06:22:28,399 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 06:22:28,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.204.212:54373/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 06:22:28,640 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 06:22:29.083467 32563 mini_ranger_kms.cc:310] Created kms service
I20260810 06:22:29.280498 32563 mini_ranger_kms.cc:326] Created kudu user
I20260810 06:22:29.325284 32563 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 06:22:29 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342947, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
I20260810 06:22:29.696276 32563 mini_ranger_kms.cc:384] Added ranger policy
I20260810 06:22:29.696422 32563 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ranger-kms:/tmp/dist-test-task7DS5L1/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7DS5L1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 06:22:29.696444 32563 mini_ranger_kms.cc:240] Using host: 127.31.204.212
I20260810 06:22:29.701565 32563 mini_ranger_kms.cc:276] Ranger KMS PID: 20238
I20260810 06:22:29.701632 32563 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.204.212:33733
Aug 10, 2026 6:22:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:22:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:22:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 6:22:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 6:22:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 6:22:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 06:22:31 dist-test-slave-q9h9 krb5kdc[19576](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786342951, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.204.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:32.184005 32563 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.982s user 0.013s sys 0.130s
I20260810 06:22:32.184142 32563 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 06:22:32.184230 32563 mini_ranger_kms.cc:401] 127.31.204.212:33733/kms/v1/keys
Aug 10 06:22:32 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342951, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.204.212@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
Aug 10 06:22:32 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342951, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.204.212@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
Aug 10 06:22:35 dist-test-slave-q9h9 krb5kdc[19576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:22:35.094321 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:45565
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:45565
--ranger_config_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.204.212:33733/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:22:35.203590 20318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:35.203815 20318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:35.203851 20318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:35.207733 20318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:22:35.207782 20318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:35.207798 20318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:35.207813 20318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:22:35.207828 20318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:22:35.212986 20318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.204.212:33733/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:45565
--ranger_config_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:45565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:35.213904 20318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:35.214892 20318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:22:35.222990 20325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:35.223003 20326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:35.223220 20318 server_base.cc:1034] running on GCE node
W20260810 06:22:35.223363 20328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:35.223858 20318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:35.224845 20318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:35.226042 20318 hybrid_clock.cc:648] HybridClock initialized: now 1786342955226003 us; error 29 us; skew 500 ppm
Aug 10 06:22:35 dist-test-slave-q9h9 krb5kdc[19576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342955, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:35.228665 20318 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:22:35.229513 20318 webserver.cc:460] Webserver started at http://127.31.204.254:35813/ using document root <none> and password file <none>
I20260810 06:22:35.229987 20318 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:35.230075 20318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:35.230273 20318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 06:22:35 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342955, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
I20260810 06:22:35.283558 20318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "893c17b1e4c2419b824b0410f24f9888"
format_stamp: "Formatted at 2026-08-10 06:22:35 on dist-test-slave-q9h9"
server_key: "01782df1571c797cc8a4f5a14cb974e8"
server_key_iv: "d55aada33e93d097d0eba474fdfc21da"
server_key_version: "kuduclusterkey@0"
I20260810 06:22:35.284327 20318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "893c17b1e4c2419b824b0410f24f9888"
format_stamp: "Formatted at 2026-08-10 06:22:35 on dist-test-slave-q9h9"
server_key: "01782df1571c797cc8a4f5a14cb974e8"
server_key_iv: "d55aada33e93d097d0eba474fdfc21da"
server_key_version: "kuduclusterkey@0"
I20260810 06:22:35.289320 20318 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 06:22:35.305612 20337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:35.307232 20318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:22:35.307372 20318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "893c17b1e4c2419b824b0410f24f9888"
format_stamp: "Formatted at 2026-08-10 06:22:35 on dist-test-slave-q9h9"
server_key: "01782df1571c797cc8a4f5a14cb974e8"
server_key_iv: "d55aada33e93d097d0eba474fdfc21da"
server_key_version: "kuduclusterkey@0"
I20260810 06:22:35.307505 20318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:35.333952 20318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:35.336925 20318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:35.337100 20318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:35.345546 20318 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:45565
I20260810 06:22:35.345568 20423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:45565 every 8 connection(s)
I20260810 06:22:35.346698 20318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:22:35.349584 20429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:22:35.350855 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 20318
I20260810 06:22:35.351004 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:22:35.351384 32563 external_mini_cluster.cc:1428] Setting key 2b5207db7d365356e28edf8b66935ec2
I20260810 06:22:35.356307 20429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888: Bootstrap starting.
I20260810 06:22:35.358775 20429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888: Neither blocks nor log segments found. Creating new log.
I20260810 06:22:35.359525 20429 log.cc:824] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888: Log is configured to *not* fsync() on all Append() calls
I20260810 06:22:35.361914 20429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888: No bootstrap required, opened a new log
Aug 10 06:22:35 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:22:35.365765 20429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893c17b1e4c2419b824b0410f24f9888" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45565 } }
I20260810 06:22:35.366189 20429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:22:35.366281 20429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893c17b1e4c2419b824b0410f24f9888, State: Initialized, Role: FOLLOWER
I20260810 06:22:35.366794 20429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893c17b1e4c2419b824b0410f24f9888" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45565 } }
I20260810 06:22:35.366953 20429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:22:35.367156 20429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:22:35.367264 20429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:22:35.368342 20429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893c17b1e4c2419b824b0410f24f9888" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45565 } }
I20260810 06:22:35.368698 20429 leader_election.cc:302] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 893c17b1e4c2419b824b0410f24f9888; no voters:
I20260810 06:22:35.369031 20429 leader_election.cc:288] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:22:35.369159 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:35.352883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45234 (local address 127.31.204.254:45565)
0810 06:22:35.353372 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:35.353389 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:35.353432 (+ 43us) server_negotiation.cc:407] Connection header received
0810 06:22:35.354286 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:35.354310 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:35.354747 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:35.355151 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:35.356167 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:35.357692 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:35.358984 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:35.359715 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:35.359872 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:35.363748 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:35.363762 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:35.363780 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:35.363814 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:35.365821 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:35.366422 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:35.366429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:35.366434 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:35.366584 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:35.366917 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:35.366921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:35.366922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:35.367448 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:35.367676 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:35.367926 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 06:22:35.368212 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":163,"threads_started":1}
I20260810 06:22:35.369176 20439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:22:35.369647 20439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [term 1 LEADER]: Becoming Leader. State: Replica: 893c17b1e4c2419b824b0410f24f9888, State: Running, Role: LEADER
I20260810 06:22:35.369935 20439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893c17b1e4c2419b824b0410f24f9888" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45565 } }
I20260810 06:22:35.370407 20429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:22:35.371598 20440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "893c17b1e4c2419b824b0410f24f9888" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893c17b1e4c2419b824b0410f24f9888" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45565 } } }
I20260810 06:22:35.371769 20440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [sys.catalog]: This master's current role is: LEADER
I20260810 06:22:35.372778 20442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 893c17b1e4c2419b824b0410f24f9888. Latest consensus state: current_term: 1 leader_uuid: "893c17b1e4c2419b824b0410f24f9888" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893c17b1e4c2419b824b0410f24f9888" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45565 } } }
I20260810 06:22:35.373389 20442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [sys.catalog]: This master's current role is: LEADER
I20260810 06:22:35.374251 20446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:22:35.374347 20429 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 06:22:35.380208 20446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:22:35.392220 20446 catalog_manager.cc:1347] Generated new cluster ID: bcbf88523e9d4bff8d3857b6e5a203fa
I20260810 06:22:35.392283 20446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:22:35.402275 20446 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:22:35.403539 20446 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:22:35.414917 20446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888: Generated new TSK 0
I20260810 06:22:35.415747 20446 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
I20260810 06:22:36.143989 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.204.212:33733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:45565
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:22:36.266551 20501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:36.266790 20501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:36.266865 20501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:36.271030 20501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:36.271083 20501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:36.271158 20501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:22:36.276508 20501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.204.212:33733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:45565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:36.277649 20501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:36.278738 20501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:22:36.289597 20511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:36.289682 20507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:36.289878 20508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:36.290226 20501 server_base.cc:1034] running on GCE node
I20260810 06:22:36.290685 20501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:36.291476 20501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:36.292691 20501 hybrid_clock.cc:648] HybridClock initialized: now 1786342956292667 us; error 32 us; skew 500 ppm
Aug 10 06:22:36 dist-test-slave-q9h9 krb5kdc[19576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342956, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:36.295701 20501 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:22:36.296530 20501 webserver.cc:460] Webserver started at http://127.31.204.193:38945/ using document root <none> and password file <none>
I20260810 06:22:36.297070 20501 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:36.297137 20501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:36.297364 20501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 06:22:36 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342956, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
I20260810 06:22:36.313393 20501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "9841360cf4ce44e0a76b1a0743305136"
format_stamp: "Formatted at 2026-08-10 06:22:36 on dist-test-slave-q9h9"
server_key: "79ad4ffd6857e9fd95b9fb8df2242379"
server_key_iv: "2a29fc27f77b07d6c7987ca1d08d742f"
server_key_version: "kuduclusterkey@0"
I20260810 06:22:36.313998 20501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "9841360cf4ce44e0a76b1a0743305136"
format_stamp: "Formatted at 2026-08-10 06:22:36 on dist-test-slave-q9h9"
server_key: "79ad4ffd6857e9fd95b9fb8df2242379"
server_key_iv: "2a29fc27f77b07d6c7987ca1d08d742f"
server_key_version: "kuduclusterkey@0"
I20260810 06:22:36.317893 20501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260810 06:22:36.332060 20522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:36.333350 20501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260810 06:22:36.333465 20501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "9841360cf4ce44e0a76b1a0743305136"
format_stamp: "Formatted at 2026-08-10 06:22:36 on dist-test-slave-q9h9"
server_key: "79ad4ffd6857e9fd95b9fb8df2242379"
server_key_iv: "2a29fc27f77b07d6c7987ca1d08d742f"
server_key_version: "kuduclusterkey@0"
I20260810 06:22:36.333580 20501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:36.375254 20501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:36.378818 20501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:36.379056 20501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:36.379829 20501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:22:36.380919 20501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:22:36.381029 20501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:36.381155 20501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:22:36.381240 20501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:36.391716 20501 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:33457
I20260810 06:22:36.392908 20501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:22:36.393851 20722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:33457 every 8 connection(s)
I20260810 06:22:36.402176 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 20501
I20260810 06:22:36.402297 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:22:36.402640 32563 external_mini_cluster.cc:1428] Setting key 538765d7427dc3d7bf93d1a7d80e0953
Aug 10 06:22:36 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342956, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:22:36.408694 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:36.396005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:51471 (local address 127.31.204.254:45565)
0810 06:22:36.396300 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:36.396309 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:36.397066 (+ 757us) server_negotiation.cc:407] Connection header received
0810 06:22:36.398049 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:36.398052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:36.398115 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:36.398202 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:36.399360 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:36.400037 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:36.401160 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:36.401431 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:36.401521 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:36.404304 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:36.404308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:36.404311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:36.404340 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:36.406217 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:36.406862 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:36.406866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:36.406868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:36.406961 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:36.407387 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:36.407390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:36.407392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:36.407618 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:36.407693 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:36.408383 (+ 690us) server_negotiation.cc:299] Negotiation successful
0810 06:22:36.408515 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":111,"threads_started":1}
I20260810 06:22:36.410472 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:36.396440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:45565 (local address 127.31.204.193:51471)
0810 06:22:36.396911 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:36.396964 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:36.397814 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:36.398387 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:36.398393 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:36.398649 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:36.399184 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:36.399200 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:36.400172 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:36.400175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:36.401041 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:36.401050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:36.401572 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:36.401576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:36.401777 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:36.402408 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:36.402428 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:36.404103 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:36.406360 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:36.406365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:36.406380 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:36.406723 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:36.407070 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:36.407073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:36.407075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:36.407263 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:36.407782 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:36.407788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:36.408099 (+ 311us) client_negotiation.cc:769] Sending connection context
0810 06:22:36.409142 (+ 1043us) client_negotiation.cc:240] Negotiation successful
0810 06:22:36.409460 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
I20260810 06:22:36.412061 20723 heartbeater.cc:347] Connected to a master server at 127.31.204.254:45565
I20260810 06:22:36.412308 20723 heartbeater.cc:464] Registering TS with master...
I20260810 06:22:36.412894 20723 heartbeater.cc:511] Master 127.31.204.254:45565 requested a full tablet report, sending...
I20260810 06:22:36.414831 20364 ts_manager.cc:194] Registered new tserver with Master: 9841360cf4ce44e0a76b1a0743305136 (127.31.204.193:33457)
I20260810 06:22:36.416378 20364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:51471
WARNING: no policy specified for kudu/127.31.204.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
I20260810 06:22:36.443266 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.204.212:33733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:45565
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 06:22:36 dist-test-slave-q9h9 krb5kdc[19576](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786342956, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 06:22:36.596036 20738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:36.596390 20738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:36.596455 20738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:36.602509 20738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:36.602626 20738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:36.602727 20738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:22:36.610379 20738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.204.212:33733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:45565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:36.611610 20738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:36.612666 20738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:22:36.625424 20749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:36.625712 20750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:36.625849 20753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:36.626832 20738 server_base.cc:1034] running on GCE node
I20260810 06:22:36.627378 20738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:36.628140 20738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:36.629311 20738 hybrid_clock.cc:648] HybridClock initialized: now 1786342956629243 us; error 81 us; skew 500 ppm
Aug 10 06:22:36 dist-test-slave-q9h9 krb5kdc[19576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342956, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:36.632891 20738 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:22:36.634001 20738 webserver.cc:460] Webserver started at http://127.31.204.194:32775/ using document root <none> and password file <none>
I20260810 06:22:36.634784 20738 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:36.634949 20738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:36.635247 20738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 06:22:36 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342956, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
I20260810 06:22:36.650885 20738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "5e36a7b8113a4c53b9977bb9e91f9af4"
format_stamp: "Formatted at 2026-08-10 06:22:36 on dist-test-slave-q9h9"
server_key: "2e2fbfa630ef51aacc4cffd70239e936"
server_key_iv: "08508a256427bfea727cadda10a05ef0"
server_key_version: "kuduclusterkey@0"
I20260810 06:22:36.652004 20738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "5e36a7b8113a4c53b9977bb9e91f9af4"
format_stamp: "Formatted at 2026-08-10 06:22:36 on dist-test-slave-q9h9"
server_key: "2e2fbfa630ef51aacc4cffd70239e936"
server_key_iv: "08508a256427bfea727cadda10a05ef0"
server_key_version: "kuduclusterkey@0"
I20260810 06:22:36.657519 20738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 06:22:36.678964 20760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:36.680740 20738 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 06:22:36.680889 20738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "5e36a7b8113a4c53b9977bb9e91f9af4"
format_stamp: "Formatted at 2026-08-10 06:22:36 on dist-test-slave-q9h9"
server_key: "2e2fbfa630ef51aacc4cffd70239e936"
server_key_iv: "08508a256427bfea727cadda10a05ef0"
server_key_version: "kuduclusterkey@0"
I20260810 06:22:36.681057 20738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:36.699740 20738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:36.704664 20738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:36.704924 20738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:36.705654 20738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:22:36.706792 20738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:22:36.706902 20738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:36.707002 20738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:22:36.707041 20738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:36.719185 20738 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:34161
I20260810 06:22:36.719252 20948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:34161 every 8 connection(s)
I20260810 06:22:36.720121 20738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:22:36.725026 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 20738
I20260810 06:22:36.725201 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:22:36.725487 32563 external_mini_cluster.cc:1428] Setting key 0405958c1ac57b80e666d5fd2813c31c
Aug 10 06:22:36 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342956, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:22:36.734009 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:36.721919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:40293 (local address 127.31.204.254:45565)
0810 06:22:36.722099 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:36.722104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:36.723065 (+ 961us) server_negotiation.cc:407] Connection header received
0810 06:22:36.724217 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:36.724220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:36.724266 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:36.724343 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:36.725810 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:36.726455 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:36.727885 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:36.728242 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:36.728368 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:36.730768 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:36.730771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:36.730773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:36.730796 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:36.732096 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:36.732561 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:36.732565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:36.732567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:36.732671 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:36.732992 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:36.732995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:36.732997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:36.733177 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:36.733261 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:36.733728 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 06:22:36.733852 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 06:22:36.734858 20953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:36.722329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:45565 (local address 127.31.204.194:40293)
0810 06:22:36.722849 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:36.722909 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:36.723988 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:36.724546 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:36.724565 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:36.724952 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:36.725633 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:36.725656 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:36.726580 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:36.726583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:36.727744 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:36.727757 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:36.728408 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:36.728412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:36.728595 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:36.729277 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:36.729298 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:36.730607 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:36.732197 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:36.732202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:36.732216 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:36.732468 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:36.732751 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:36.732754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:36.732756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:36.732902 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:36.733275 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:36.733282 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:36.733508 (+ 226us) client_negotiation.cc:769] Sending connection context
0810 06:22:36.733676 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 06:22:36.733964 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":104,"threads_started":1}
I20260810 06:22:36.735929 20949 heartbeater.cc:347] Connected to a master server at 127.31.204.254:45565
I20260810 06:22:36.736143 20949 heartbeater.cc:464] Registering TS with master...
I20260810 06:22:36.736595 20949 heartbeater.cc:511] Master 127.31.204.254:45565 requested a full tablet report, sending...
I20260810 06:22:36.737524 20364 ts_manager.cc:194] Registered new tserver with Master: 5e36a7b8113a4c53b9977bb9e91f9af4 (127.31.204.194:34161)
I20260810 06:22:36.738103 20364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:40293
WARNING: no policy specified for kudu/127.31.204.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
I20260810 06:22:36.762579 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.204.212:33733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:45565
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:22:36.922230 20961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:36.922606 20961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:36.922709 20961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:36.928263 20961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:36.928436 20961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:36.928583 20961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:22:36.935930 20961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.204.212:33733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:45565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:36.937326 20961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:36.938733 20961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Aug 10 06:22:36 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342956, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
W20260810 06:22:36.952140 20976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:36.952338 20972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:36.959364 20961 server_base.cc:1034] running on GCE node
W20260810 06:22:36.960162 20971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:36.960729 20961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:36.961594 20961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:36.968986 20961 hybrid_clock.cc:648] HybridClock initialized: now 1786342956967493 us; error 1500 us; skew 500 ppm
Aug 10 06:22:36 dist-test-slave-q9h9 krb5kdc[19576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342956, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:36.972846 20961 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:22:36.974210 20961 webserver.cc:460] Webserver started at http://127.31.204.195:38729/ using document root <none> and password file <none>
I20260810 06:22:36.975020 20961 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:36.975154 20961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:36.975421 20961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 06:22:36 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342956, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
I20260810 06:22:37.000322 20961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "9c4206f97a9f438ba0d23a82c999e439"
format_stamp: "Formatted at 2026-08-10 06:22:36 on dist-test-slave-q9h9"
server_key: "d5b6f4cd733f6f1fb1b2d6748ec982aa"
server_key_iv: "2263a3edd4bd37bc30435fbc5ef8b250"
server_key_version: "kuduclusterkey@0"
I20260810 06:22:37.001084 20961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "9c4206f97a9f438ba0d23a82c999e439"
format_stamp: "Formatted at 2026-08-10 06:22:36 on dist-test-slave-q9h9"
server_key: "d5b6f4cd733f6f1fb1b2d6748ec982aa"
server_key_iv: "2263a3edd4bd37bc30435fbc5ef8b250"
server_key_version: "kuduclusterkey@0"
I20260810 06:22:37.034261 20961 fs_manager.cc:691] Time spent creating directory manager: real 0.033s user 0.005s sys 0.002s
I20260810 06:22:37.070469 20987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:37.072515 20961 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260810 06:22:37.072647 20961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "9c4206f97a9f438ba0d23a82c999e439"
format_stamp: "Formatted at 2026-08-10 06:22:36 on dist-test-slave-q9h9"
server_key: "d5b6f4cd733f6f1fb1b2d6748ec982aa"
server_key_iv: "2263a3edd4bd37bc30435fbc5ef8b250"
server_key_version: "kuduclusterkey@0"
I20260810 06:22:37.072789 20961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:37.091435 20961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:37.094734 20961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:37.094930 20961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:37.095516 20961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:22:37.096459 20961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:22:37.096509 20961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:37.096562 20961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:22:37.096617 20961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:37.110836 20961 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:33429
I20260810 06:22:37.110883 21182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:33429 every 8 connection(s)
I20260810 06:22:37.111852 20961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:22:37.112531 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 20961
I20260810 06:22:37.112679 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:22:37.113070 32563 external_mini_cluster.cc:1428] Setting key ff9cdee7591545359b98fc5ea4e3a880
Aug 10 06:22:37 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342956, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:22:37.127094 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:37.117117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:41077 (local address 127.31.204.254:45565)
0810 06:22:37.117238 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:37.117242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:37.117259 (+ 17us) server_negotiation.cc:407] Connection header received
0810 06:22:37.117830 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:37.117833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:37.117893 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:37.117974 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:37.119084 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:37.119629 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:37.120782 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:37.121022 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:37.121115 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:37.123554 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:37.123559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:37.123562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:37.123587 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:37.124991 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:37.125497 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:37.125501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:37.125503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:37.125588 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:37.125925 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:37.125929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:37.125930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:37.126164 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:37.126267 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:37.126764 (+ 497us) server_negotiation.cc:299] Negotiation successful
0810 06:22:37.126900 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:22:37.128620 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:37.115895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:45565 (local address 127.31.204.195:41077)
0810 06:22:37.116595 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:37.116650 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:37.117626 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:37.118185 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:37.118192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:37.118468 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:37.118935 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:37.118950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:37.119759 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:37.119763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:37.120655 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:37.120666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:37.121176 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:37.121181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:37.121363 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:37.121941 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:37.121959 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:37.123375 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:37.125111 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:37.125117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:37.125131 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:37.125395 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:37.125678 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:37.125681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:37.125683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:37.125831 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:37.126268 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:37.126275 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:37.126513 (+ 238us) client_negotiation.cc:769] Sending connection context
0810 06:22:37.126700 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 06:22:37.127001 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":288,"threads_started":1}
I20260810 06:22:37.129915 21184 heartbeater.cc:347] Connected to a master server at 127.31.204.254:45565
I20260810 06:22:37.130190 21184 heartbeater.cc:464] Registering TS with master...
I20260810 06:22:37.130673 21184 heartbeater.cc:511] Master 127.31.204.254:45565 requested a full tablet report, sending...
I20260810 06:22:37.131690 20364 ts_manager.cc:194] Registered new tserver with Master: 9c4206f97a9f438ba0d23a82c999e439 (127.31.204.195:33429)
I20260810 06:22:37.132251 20364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:41077
I20260810 06:22:37.140702 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:22:37.141383 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 20501
I20260810 06:22:37.164526 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 20738
I20260810 06:22:37.189708 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 20961
I20260810 06:22:37.203282 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 20318
I20260810 06:22:37.217182 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:45565
--webserver_interface=127.31.204.254
--webserver_port=35813
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:45565
--ranger_config_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.204.212:33733/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.204.212:33733/kms with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:22:37.327615 21188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:37.327857 21188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:37.327896 21188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:37.331743 21188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:22:37.331794 21188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:37.331815 21188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:37.331832 21188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:22:37.331849 21188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:22:37.337109 21188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.204.212:33733/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:45565
--ranger_config_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:45565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=35813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:37.338001 21188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:37.339002 21188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:22:37.347091 21198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:37.347102 21194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:37.347126 21196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:37.347232 21188 server_base.cc:1034] running on GCE node
I20260810 06:22:37.347908 21188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:37.348937 21188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:37.350077 21188 hybrid_clock.cc:648] HybridClock initialized: now 1786342957350061 us; error 29 us; skew 500 ppm
Aug 10 06:22:37 dist-test-slave-q9h9 krb5kdc[19576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342957, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:37.356220 21188 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:22:37.357056 21188 webserver.cc:460] Webserver started at http://127.31.204.254:35813/ using document root <none> and password file <none>
I20260810 06:22:37.357555 21188 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:37.357641 21188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 06:22:37 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342957, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
I20260810 06:22:37.403263 21188 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 06:22:37.405769 21210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:37.407270 21188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 06:22:37.407388 21188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "893c17b1e4c2419b824b0410f24f9888"
format_stamp: "Formatted at 2026-08-10 06:22:35 on dist-test-slave-q9h9"
server_key: "01782df1571c797cc8a4f5a14cb974e8"
server_key_iv: "d55aada33e93d097d0eba474fdfc21da"
server_key_version: "kuduclusterkey@0"
I20260810 06:22:37.407936 21188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:37.425168 21188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:37.427927 21188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:37.428093 21188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:37.434983 21188 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:45565
I20260810 06:22:37.434998 21284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:45565 every 8 connection(s)
I20260810 06:22:37.435988 21188 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:22:37.440032 21286 sys_catalog.cc:263] Verifying existing consensus state
I20260810 06:22:37.441572 21286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888: Bootstrap starting.
I20260810 06:22:37.444231 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 21188
I20260810 06:22:37.444664 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.204.212:33733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:33457
--local_ip_for_outbound_sockets=127.31.204.193
--tserver_master_addrs=127.31.204.254:45565
--webserver_port=38945
--webserver_interface=127.31.204.193
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.204.212:33733/kms with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:22:37.457695 21286 log.cc:824] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888: Log is configured to *not* fsync() on all Append() calls
I20260810 06:22:37.464170 21286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:22:37.464511 21286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888: Bootstrap complete.
I20260810 06:22:37.468297 21286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893c17b1e4c2419b824b0410f24f9888" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45565 } }
I20260810 06:22:37.468662 21286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893c17b1e4c2419b824b0410f24f9888, State: Initialized, Role: FOLLOWER
I20260810 06:22:37.469134 21286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893c17b1e4c2419b824b0410f24f9888" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45565 } }
I20260810 06:22:37.469260 21286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:22:37.469357 21286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:22:37.469437 21286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [term 1 FOLLOWER]: Advancing to term 2
I20260810 06:22:37.471254 21286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893c17b1e4c2419b824b0410f24f9888" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45565 } }
I20260810 06:22:37.471555 21286 leader_election.cc:302] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 893c17b1e4c2419b824b0410f24f9888; no voters:
I20260810 06:22:37.471840 21286 leader_election.cc:288] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 06:22:37.471962 21291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 06:22:37.472198 21291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [term 2 LEADER]: Becoming Leader. State: Replica: 893c17b1e4c2419b824b0410f24f9888, State: Running, Role: LEADER
I20260810 06:22:37.472548 21291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893c17b1e4c2419b824b0410f24f9888" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45565 } }
I20260810 06:22:37.472924 21286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:22:37.473874 21292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "893c17b1e4c2419b824b0410f24f9888" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893c17b1e4c2419b824b0410f24f9888" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45565 } } }
I20260810 06:22:37.473994 21292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [sys.catalog]: This master's current role is: LEADER
I20260810 06:22:37.474926 21296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 893c17b1e4c2419b824b0410f24f9888. Latest consensus state: current_term: 2 leader_uuid: "893c17b1e4c2419b824b0410f24f9888" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893c17b1e4c2419b824b0410f24f9888" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45565 } } }
I20260810 06:22:37.475024 21296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888 [sys.catalog]: This master's current role is: LEADER
I20260810 06:22:37.475100 21286 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 06:22:37.475276 21307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:22:37.481411 21307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:22:37.482244 21307 catalog_manager.cc:1259] Loaded cluster ID: bcbf88523e9d4bff8d3857b6e5a203fa
I20260810 06:22:37.482358 21307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:22:37.484848 21307 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:22:37.485590 21307 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888: Loaded TSK: 0
I20260810 06:22:37.486876 21307 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 06:22:37.581398 21288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:37.581818 21288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:37.581938 21288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:37.586843 21288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:37.586900 21288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:37.586977 21288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:22:37.592114 21288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.204.212:33733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:33457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=38945
--tserver_master_addrs=127.31.204.254:45565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:37.593039 21288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:37.594103 21288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:22:37.603931 21355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:37.603889 21352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:37.604202 21353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:37.604700 21288 server_base.cc:1034] running on GCE node
I20260810 06:22:37.605159 21288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:37.605857 21288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:37.607072 21288 hybrid_clock.cc:648] HybridClock initialized: now 1786342957607049 us; error 35 us; skew 500 ppm
Aug 10 06:22:37 dist-test-slave-q9h9 krb5kdc[19576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342957, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:37.610338 21288 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:22:37.611383 21288 webserver.cc:460] Webserver started at http://127.31.204.193:38945/ using document root <none> and password file <none>
I20260810 06:22:37.612008 21288 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:37.612246 21288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 06:22:37 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342957, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
I20260810 06:22:37.647455 21288 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:22:37.649999 21365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:37.651294 21288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:22:37.651401 21288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "9841360cf4ce44e0a76b1a0743305136"
format_stamp: "Formatted at 2026-08-10 06:22:36 on dist-test-slave-q9h9"
server_key: "79ad4ffd6857e9fd95b9fb8df2242379"
server_key_iv: "2a29fc27f77b07d6c7987ca1d08d742f"
server_key_version: "kuduclusterkey@0"
I20260810 06:22:37.651926 21288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:37.705262 21288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:37.708829 21288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:37.709084 21288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:37.709791 21288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:22:37.710832 21288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:22:37.710891 21288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:37.710942 21288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:22:37.711021 21288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260810 06:22:37.721206 21288 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:33457
I20260810 06:22:37.721283 21551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:33457 every 8 connection(s)
I20260810 06:22:37.722442 21288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:22:37.726785 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 21288
I20260810 06:22:37.727190 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.204.212:33733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:34161
--local_ip_for_outbound_sockets=127.31.204.194
--tserver_master_addrs=127.31.204.254:45565
--webserver_port=32775
--webserver_interface=127.31.204.194
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.204.212:33733/kms with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 06:22:37 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342957, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:22:37.743245 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:37.724500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:35425 (local address 127.31.204.254:45565)
0810 06:22:37.728018 (+ 3518us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:37.728041 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:37.728086 (+ 45us) server_negotiation.cc:407] Connection header received
0810 06:22:37.728311 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:37.728321 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:37.728598 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:37.728901 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:37.730189 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:37.731730 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:37.733433 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:37.734155 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:37.734305 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:37.737466 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:37.737473 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:37.737490 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:37.737525 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:37.739563 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:37.740197 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:37.740204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:37.740209 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:37.740365 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:37.740771 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:37.740775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:37.740777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:37.741203 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:37.741317 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:37.741987 (+ 670us) server_negotiation.cc:299] Negotiation successful
0810 06:22:37.742300 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3348,"thread_start_us":142,"threads_started":1}
I20260810 06:22:37.745040 21560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:37.724532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:45565 (local address 127.31.204.193:35425)
0810 06:22:37.724989 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:37.725038 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:37.726003 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:37.729002 (+ 2999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:37.729010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:37.729359 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:37.729960 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:37.729979 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:37.731904 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:37.731907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:37.733286 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:37.733298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:37.734623 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:37.734629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:37.734850 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:37.735498 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:37.735518 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:37.737236 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:37.739703 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:37.739709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:37.739722 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:37.740053 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:37.740493 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:37.740496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:37.740499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:37.740663 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:37.741402 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:37.741407 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:37.741666 (+ 259us) client_negotiation.cc:769] Sending connection context
0810 06:22:37.743923 (+ 2257us) client_negotiation.cc:240] Negotiation successful
0810 06:22:37.744210 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":118,"threads_started":1}
I20260810 06:22:37.748876 21552 heartbeater.cc:347] Connected to a master server at 127.31.204.254:45565
I20260810 06:22:37.749087 21552 heartbeater.cc:464] Registering TS with master...
I20260810 06:22:37.749598 21552 heartbeater.cc:511] Master 127.31.204.254:45565 requested a full tablet report, sending...
W20260810 06:22:37.751729 21552 heartbeater.cc:644] Failed to heartbeat to 127.31.204.254:45565 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 06:22:37.751837 21552 heartbeater.cc:464] Registering TS with master...
I20260810 06:22:37.751981 21552 heartbeater.cc:511] Master 127.31.204.254:45565 requested a full tablet report, sending...
I20260810 06:22:37.752785 21552 heartbeater.cc:464] Registering TS with master...
I20260810 06:22:37.752936 21552 heartbeater.cc:511] Master 127.31.204.254:45565 requested a full tablet report, sending...
W20260810 06:22:37.753655 21552 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 06:22:37.859040 21563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:37.859283 21563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:37.859339 21563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:37.864848 21563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:37.864974 21563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:37.865097 21563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:22:37.872241 21563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.204.212:33733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:34161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=32775
--tserver_master_addrs=127.31.204.254:45565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:37.873664 21563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:37.875041 21563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:22:37.885509 21569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:37.885969 21563 server_base.cc:1034] running on GCE node
W20260810 06:22:37.886121 21571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:37.886242 21576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:37.886643 21563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:37.887247 21563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:37.888434 21563 hybrid_clock.cc:648] HybridClock initialized: now 1786342957888408 us; error 37 us; skew 500 ppm
Aug 10 06:22:37 dist-test-slave-q9h9 krb5kdc[19576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342957, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:37.891901 21563 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:22:37.892989 21563 webserver.cc:460] Webserver started at http://127.31.204.194:32775/ using document root <none> and password file <none>
I20260810 06:22:37.893705 21563 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:37.893877 21563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 06:22:37 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342957, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
I20260810 06:22:37.919302 21563 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.000s sys 0.002s
I20260810 06:22:37.922487 21592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:37.923995 21563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260810 06:22:37.924108 21563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "5e36a7b8113a4c53b9977bb9e91f9af4"
format_stamp: "Formatted at 2026-08-10 06:22:36 on dist-test-slave-q9h9"
server_key: "2e2fbfa630ef51aacc4cffd70239e936"
server_key_iv: "08508a256427bfea727cadda10a05ef0"
server_key_version: "kuduclusterkey@0"
I20260810 06:22:37.924787 21563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:37.945891 21563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:37.948621 21563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:37.948800 21563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:37.949524 21563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:22:37.950632 21563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:22:37.950742 21563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:37.950861 21563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:22:37.950937 21563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:37.961691 21563 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:34161
I20260810 06:22:37.962698 21782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:34161 every 8 connection(s)
I20260810 06:22:37.963207 21563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:22:37.972431 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 21563
I20260810 06:22:37.972856 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.204.212:33733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:33429
--local_ip_for_outbound_sockets=127.31.204.195
--tserver_master_addrs=127.31.204.254:45565
--webserver_port=38729
--webserver_interface=127.31.204.195
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.204.212:33733/kms with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 06:22:37 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342957, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:22:37.978943 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:37.965518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:54459 (local address 127.31.204.254:45565)
0810 06:22:37.965683 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:37.965688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:37.966569 (+ 881us) server_negotiation.cc:407] Connection header received
0810 06:22:37.967725 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:37.967728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:37.967783 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:37.967850 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:37.969200 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:37.969699 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:37.971321 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:37.971535 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:37.971607 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:37.974531 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:37.974536 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:37.974539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:37.974569 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:37.976492 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:37.977132 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:37.977135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:37.977136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:37.977215 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:37.977606 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:37.977610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:37.977611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:37.977842 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:37.977940 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:37.978643 (+ 703us) server_negotiation.cc:299] Negotiation successful
0810 06:22:37.978776 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 06:22:37.979892 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:37.965864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:45565 (local address 127.31.204.194:54459)
0810 06:22:37.966392 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:37.966450 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:37.967507 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:37.968044 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:37.968054 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:37.968410 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:37.969059 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:37.969082 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:37.969828 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:37.969831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:37.971196 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:37.971208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:37.971688 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:37.971693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:37.971902 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:37.972549 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:37.972572 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:37.974320 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:37.976618 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:37.976624 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:37.976638 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:37.976982 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:37.977300 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:37.977304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:37.977306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:37.977479 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:37.977994 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:37.978000 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:37.978323 (+ 323us) client_negotiation.cc:769] Sending connection context
0810 06:22:37.978543 (+ 220us) client_negotiation.cc:240] Negotiation successful
0810 06:22:37.978883 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":137,"threads_started":1}
I20260810 06:22:37.981469 21783 heartbeater.cc:347] Connected to a master server at 127.31.204.254:45565
I20260810 06:22:37.981721 21783 heartbeater.cc:464] Registering TS with master...
I20260810 06:22:37.984395 21783 heartbeater.cc:511] Master 127.31.204.254:45565 requested a full tablet report, sending...
W20260810 06:22:37.986056 21783 heartbeater.cc:644] Failed to heartbeat to 127.31.204.254:45565 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 06:22:37.986188 21783 heartbeater.cc:464] Registering TS with master...
I20260810 06:22:37.986362 21783 heartbeater.cc:511] Master 127.31.204.254:45565 requested a full tablet report, sending...
I20260810 06:22:37.990787 21783 heartbeater.cc:464] Registering TS with master...
I20260810 06:22:37.990962 21783 heartbeater.cc:511] Master 127.31.204.254:45565 requested a full tablet report, sending...
W20260810 06:22:37.991822 21783 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 06:22:38.140813 21787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:38.141165 21787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:38.141238 21787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:38.147270 21787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:38.147353 21787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:38.147441 21787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:22:38.155120 21787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.204.212:33733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:33429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=38729
--tserver_master_addrs=127.31.204.254:45565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:38.156554 21787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:38.157960 21787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:22:38.169287 21799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:38.169437 21787 server_base.cc:1034] running on GCE node
W20260810 06:22:38.169328 21801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:38.169287 21803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:38.169993 21787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:38.170625 21787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:38.171790 21787 hybrid_clock.cc:648] HybridClock initialized: now 1786342958171676 us; error 128 us; skew 500 ppm
Aug 10 06:22:38 dist-test-slave-q9h9 krb5kdc[19576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342958, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:38.178912 21787 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:22:38.180083 21787 webserver.cc:460] Webserver started at http://127.31.204.195:38729/ using document root <none> and password file <none>
I20260810 06:22:38.180806 21787 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:38.180914 21787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 06:22:38 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342958, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
I20260810 06:22:38.207567 21787 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.000s sys 0.003s
I20260810 06:22:38.211329 21818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:38.212909 21787 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260810 06:22:38.213047 21787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "9c4206f97a9f438ba0d23a82c999e439"
format_stamp: "Formatted at 2026-08-10 06:22:36 on dist-test-slave-q9h9"
server_key: "d5b6f4cd733f6f1fb1b2d6748ec982aa"
server_key_iv: "2263a3edd4bd37bc30435fbc5ef8b250"
server_key_version: "kuduclusterkey@0"
I20260810 06:22:38.213795 21787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:38.236258 21787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:38.240005 21787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:38.240229 21787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:38.241008 21787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:22:38.242149 21787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:22:38.242214 21787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:38.242270 21787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:22:38.242300 21787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:38.253584 21787 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:33429
I20260810 06:22:38.253598 22014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:33429 every 8 connection(s)
I20260810 06:22:38.254868 21787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:22:38.262825 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 21787
Aug 10 06:22:38 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342958, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:22:38.271420 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:38.256762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:49265 (local address 127.31.204.254:45565)
0810 06:22:38.256875 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:38.256878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:38.258062 (+ 1184us) server_negotiation.cc:407] Connection header received
0810 06:22:38.259302 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:38.259307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:38.259367 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:38.259457 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:38.260879 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:38.261677 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:38.263418 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:38.263765 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:38.263840 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:38.267415 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:38.267418 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:38.267421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:38.267444 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:38.269030 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:38.269596 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:38.269599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:38.269600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:38.269676 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:38.270367 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:38.270371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:38.270372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:38.270545 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:38.270629 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:38.271135 (+ 506us) server_negotiation.cc:299] Negotiation successful
0810 06:22:38.271256 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 06:22:38.272266 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:38.257304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:45565 (local address 127.31.204.195:49265)
0810 06:22:38.257872 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:38.257928 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:38.259033 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:38.259679 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:38.259687 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:38.260037 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:38.260686 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:38.260705 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:38.261847 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:38.261851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:38.263269 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:38.263281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:38.264041 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:38.264045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:38.264257 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:38.264951 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:38.264973 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:38.267250 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:38.269133 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:38.269140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:38.269156 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:38.269494 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:38.270104 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:38.270107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:38.270109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:38.270254 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:38.270642 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:38.270649 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:38.270897 (+ 248us) client_negotiation.cc:769] Sending connection context
0810 06:22:38.271060 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 06:22:38.271385 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":143,"threads_started":1}
I20260810 06:22:38.273517 22017 heartbeater.cc:347] Connected to a master server at 127.31.204.254:45565
I20260810 06:22:38.273787 22017 heartbeater.cc:464] Registering TS with master...
I20260810 06:22:38.274448 22017 heartbeater.cc:511] Master 127.31.204.254:45565 requested a full tablet report, sending...
I20260810 06:22:38.274570 22022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:38.263865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58454 (local address 127.31.204.254:45565)
0810 06:22:38.264121 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:38.264128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:38.264138 (+ 10us) server_negotiation.cc:407] Connection header received
0810 06:22:38.264167 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:38.264169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:38.264202 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:38.264276 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:38.265136 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:38.265643 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:38.266902 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:38.267123 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:38.267215 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:38.268747 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:38.268751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:38.268753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:38.268804 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:38.270589 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:38.273072 (+ 2483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:38.273076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:38.273077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:38.273174 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:38.273466 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:38.273469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:38.273471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:38.273673 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:38.273897 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:38.274281 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 06:22:38.274407 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":99,"threads_started":1}
I20260810 06:22:38.275960 21236 ts_manager.cc:194] Registered new tserver with Master: 9c4206f97a9f438ba0d23a82c999e439 (127.31.204.195:33429)
I20260810 06:22:38.277590 21236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:49265
Aug 10 06:22:38 dist-test-slave-q9h9 krb5kdc[19576](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786342958, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:38.753868 21552 heartbeater.cc:464] Registering TS with master...
I20260810 06:22:38.754163 21552 heartbeater.cc:511] Master 127.31.204.254:45565 requested a full tablet report, sending...
I20260810 06:22:38.754832 21236 ts_manager.cc:194] Registered new tserver with Master: 9841360cf4ce44e0a76b1a0743305136 (127.31.204.193:33457)
I20260810 06:22:38.755417 21236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:35425
Aug 10 06:22:38 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342958, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
I20260810 06:22:38.992050 21783 heartbeater.cc:464] Registering TS with master...
I20260810 06:22:38.992341 21783 heartbeater.cc:511] Master 127.31.204.254:45565 requested a full tablet report, sending...
I20260810 06:22:38.993052 21236 ts_manager.cc:194] Registered new tserver with Master: 5e36a7b8113a4c53b9977bb9e91f9af4 (127.31.204.194:34161)
I20260810 06:22:38.993857 21236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:54459
I20260810 06:22:38.994051 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:22:39.006052 22040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:38.995750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58464 (local address 127.31.204.254:45565)
0810 06:22:38.996369 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:38.996374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:38.996776 (+ 402us) server_negotiation.cc:407] Connection header received
0810 06:22:38.996821 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:38.996823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:38.996871 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:38.996938 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:38.997846 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:38.998683 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:38.999874 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.000240 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.000336 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:39.002182 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:39.002186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:39.002189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:39.002216 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:39.003871 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:39.004354 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:39.004357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:39.004359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:39.004448 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:39.004742 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:39.004745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:39.004747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:39.004948 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:39.005069 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:39.005724 (+ 655us) server_negotiation.cc:299] Negotiation successful
0810 06:22:39.005852 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":101,"threads_started":1}
I20260810 06:22:39.010509 21236 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:22:39.015337 21236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:22:39.047703 22052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:39.027943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:34161 (local address 127.0.0.1:56994)
0810 06:22:39.030793 (+ 2850us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:39.030848 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:39.030969 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:39.033802 (+ 2833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:39.033808 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:39.033835 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:22:39.033971 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:39.033978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.043917 (+ 9939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.043920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:39.045795 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:39.045807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.047107 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.047111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:39.047222 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:39.047268 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 06:22:39.047388 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 06:22:39.047524 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":100,"threads_started":1}
I20260810 06:22:39.053128 22062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:39.029385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:33429 (local address 127.0.0.1:46080)
0810 06:22:39.036006 (+ 6621us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:39.036021 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:39.036112 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:39.037465 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:39.037468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:39.037485 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:22:39.037581 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:39.037587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.039952 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.039955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:39.042272 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:39.042284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.052729 (+ 10445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.052732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:39.052829 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:39.052853 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 06:22:39.052920 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 06:22:39.052985 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6552,"thread_start_us":91,"threads_started":1}
I20260810 06:22:39.053722 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:39.029801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46080 (local address 127.31.204.195:33429)
0810 06:22:39.030092 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:39.030099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:39.036210 (+ 6111us) server_negotiation.cc:407] Connection header received
0810 06:22:39.036293 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:39.036299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:39.037182 (+ 883us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:39.037327 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:22:39.037857 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.039240 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.042421 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.043272 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.043395 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:39.043537 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:39.053440 (+ 9903us) server_negotiation.cc:299] Negotiation successful
0810 06:22:39.053562 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":74,"threads_started":1}
I20260810 06:22:39.054507 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:39.028112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.31.204.194:34161)
0810 06:22:39.031579 (+ 3467us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:39.031586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:39.031611 (+ 25us) server_negotiation.cc:407] Connection header received
0810 06:22:39.031702 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:39.031707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:39.032085 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:39.032373 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:22:39.034705 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.043782 (+ 9077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.045944 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.046787 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.046909 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:39.047039 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:39.054262 (+ 7223us) server_negotiation.cc:299] Negotiation successful
0810 06:22:39.054367 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3204,"thread_start_us":113,"threads_started":1}
I20260810 06:22:39.054626 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:39.030224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.31.204.193:33457)
0810 06:22:39.030541 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:39.030547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:39.048523 (+ 17976us) server_negotiation.cc:407] Connection header received
0810 06:22:39.048616 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:39.048621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:39.048870 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:39.049029 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:22:39.049425 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.051306 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.052723 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.053531 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.053662 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:39.053780 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:39.054378 (+ 598us) server_negotiation.cc:299] Negotiation successful
0810 06:22:39.054484 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":117,"threads_started":1}
I20260810 06:22:39.054288 22052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:39.029658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:33457 (local address 127.0.0.1:58582)
0810 06:22:39.048323 (+ 18665us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:39.048336 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:39.048412 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:39.049188 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:39.049191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:39.049211 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:22:39.049285 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:39.049291 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.051444 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.051448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:39.052568 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:39.052580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.053847 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.053850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:39.053942 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:39.053965 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 06:22:39.054097 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 06:22:39.054166 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18583,"thread_start_us":92,"threads_started":1}
I20260810 06:22:39.058002 21681 tablet_service.cc:1467] Processing CreateTablet for tablet a35a453b171b4ae48c0e9584513aff36 (DEFAULT_TABLE table=test-table [id=e6fac965444d4c74bc2710741586cc28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:22:39.058001 21882 tablet_service.cc:1467] Processing CreateTablet for tablet a35a453b171b4ae48c0e9584513aff36 (DEFAULT_TABLE table=test-table [id=e6fac965444d4c74bc2710741586cc28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:22:39.059180 21681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35a453b171b4ae48c0e9584513aff36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:22:39.058912 21445 tablet_service.cc:1467] Processing CreateTablet for tablet a35a453b171b4ae48c0e9584513aff36 (DEFAULT_TABLE table=test-table [id=e6fac965444d4c74bc2710741586cc28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:22:39.059372 21882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35a453b171b4ae48c0e9584513aff36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:22:39.059921 21445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35a453b171b4ae48c0e9584513aff36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:22:39.069082 22067 tablet_bootstrap.cc:492] T a35a453b171b4ae48c0e9584513aff36 P 9841360cf4ce44e0a76b1a0743305136: Bootstrap starting.
I20260810 06:22:39.071430 22067 tablet_bootstrap.cc:654] T a35a453b171b4ae48c0e9584513aff36 P 9841360cf4ce44e0a76b1a0743305136: Neither blocks nor log segments found. Creating new log.
I20260810 06:22:39.072297 22067 log.cc:824] T a35a453b171b4ae48c0e9584513aff36 P 9841360cf4ce44e0a76b1a0743305136: Log is configured to *not* fsync() on all Append() calls
I20260810 06:22:39.072976 22066 tablet_bootstrap.cc:492] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439: Bootstrap starting.
I20260810 06:22:39.075280 22066 tablet_bootstrap.cc:654] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439: Neither blocks nor log segments found. Creating new log.
I20260810 06:22:39.076113 22066 log.cc:824] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439: Log is configured to *not* fsync() on all Append() calls
I20260810 06:22:39.078389 22066 tablet_bootstrap.cc:492] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439: No bootstrap required, opened a new log
I20260810 06:22:39.078593 22066 ts_tablet_manager.cc:1397] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 06:22:39.078805 22070 tablet_bootstrap.cc:492] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4: Bootstrap starting.
I20260810 06:22:39.080170 22067 tablet_bootstrap.cc:492] T a35a453b171b4ae48c0e9584513aff36 P 9841360cf4ce44e0a76b1a0743305136: No bootstrap required, opened a new log
I20260810 06:22:39.080346 22067 ts_tablet_manager.cc:1397] T a35a453b171b4ae48c0e9584513aff36 P 9841360cf4ce44e0a76b1a0743305136: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20260810 06:22:39.080885 22070 tablet_bootstrap.cc:654] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4: Neither blocks nor log segments found. Creating new log.
I20260810 06:22:39.081661 22070 log.cc:824] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4: Log is configured to *not* fsync() on all Append() calls
I20260810 06:22:39.084030 22066 raft_consensus.cc:348] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36a7b8113a4c53b9977bb9e91f9af4" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34161 } } peers { permanent_uuid: "9c4206f97a9f438ba0d23a82c999e439" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 33429 } } peers { permanent_uuid: "9841360cf4ce44e0a76b1a0743305136" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 33457 } }
I20260810 06:22:39.084300 22066 raft_consensus.cc:374] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:22:39.084355 22066 raft_consensus.cc:729] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c4206f97a9f438ba0d23a82c999e439, State: Initialized, Role: FOLLOWER
I20260810 06:22:39.084167 22067 raft_consensus.cc:348] T a35a453b171b4ae48c0e9584513aff36 P 9841360cf4ce44e0a76b1a0743305136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36a7b8113a4c53b9977bb9e91f9af4" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34161 } } peers { permanent_uuid: "9c4206f97a9f438ba0d23a82c999e439" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 33429 } } peers { permanent_uuid: "9841360cf4ce44e0a76b1a0743305136" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 33457 } }
I20260810 06:22:39.084647 22067 raft_consensus.cc:374] T a35a453b171b4ae48c0e9584513aff36 P 9841360cf4ce44e0a76b1a0743305136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:22:39.084750 22067 raft_consensus.cc:729] T a35a453b171b4ae48c0e9584513aff36 P 9841360cf4ce44e0a76b1a0743305136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9841360cf4ce44e0a76b1a0743305136, State: Initialized, Role: FOLLOWER
I20260810 06:22:39.084807 22066 consensus_queue.cc:260] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36a7b8113a4c53b9977bb9e91f9af4" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34161 } } peers { permanent_uuid: "9c4206f97a9f438ba0d23a82c999e439" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 33429 } } peers { permanent_uuid: "9841360cf4ce44e0a76b1a0743305136" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 33457 } }
I20260810 06:22:39.085320 22067 consensus_queue.cc:260] T a35a453b171b4ae48c0e9584513aff36 P 9841360cf4ce44e0a76b1a0743305136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36a7b8113a4c53b9977bb9e91f9af4" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34161 } } peers { permanent_uuid: "9c4206f97a9f438ba0d23a82c999e439" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 33429 } } peers { permanent_uuid: "9841360cf4ce44e0a76b1a0743305136" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 33457 } }
I20260810 06:22:39.085868 22066 ts_tablet_manager.cc:1428] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 06:22:39.086256 22017 heartbeater.cc:503] Master 127.31.204.254:45565 was elected leader, sending a full tablet report...
I20260810 06:22:39.086470 22067 ts_tablet_manager.cc:1428] T a35a453b171b4ae48c0e9584513aff36 P 9841360cf4ce44e0a76b1a0743305136: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20260810 06:22:39.087158 21552 heartbeater.cc:503] Master 127.31.204.254:45565 was elected leader, sending a full tablet report...
I20260810 06:22:39.095916 22070 tablet_bootstrap.cc:492] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4: No bootstrap required, opened a new log
I20260810 06:22:39.096127 22070 ts_tablet_manager.cc:1397] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20260810 06:22:39.099020 22070 raft_consensus.cc:348] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36a7b8113a4c53b9977bb9e91f9af4" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34161 } } peers { permanent_uuid: "9c4206f97a9f438ba0d23a82c999e439" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 33429 } } peers { permanent_uuid: "9841360cf4ce44e0a76b1a0743305136" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 33457 } }
I20260810 06:22:39.099220 22070 raft_consensus.cc:374] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:22:39.099272 22070 raft_consensus.cc:729] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e36a7b8113a4c53b9977bb9e91f9af4, State: Initialized, Role: FOLLOWER
I20260810 06:22:39.099632 22070 consensus_queue.cc:260] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36a7b8113a4c53b9977bb9e91f9af4" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34161 } } peers { permanent_uuid: "9c4206f97a9f438ba0d23a82c999e439" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 33429 } } peers { permanent_uuid: "9841360cf4ce44e0a76b1a0743305136" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 33457 } }
I20260810 06:22:39.100239 21783 heartbeater.cc:503] Master 127.31.204.254:45565 was elected leader, sending a full tablet report...
I20260810 06:22:39.100564 22070 ts_tablet_manager.cc:1428] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
W20260810 06:22:39.215598 21784 tablet.cc:2369] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:22:39.230572 21556 tablet.cc:2369] T a35a453b171b4ae48c0e9584513aff36 P 9841360cf4ce44e0a76b1a0743305136: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:22:39.237459 22078 raft_consensus.cc:482] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:22:39.237715 22078 raft_consensus.cc:504] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36a7b8113a4c53b9977bb9e91f9af4" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34161 } } peers { permanent_uuid: "9c4206f97a9f438ba0d23a82c999e439" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 33429 } } peers { permanent_uuid: "9841360cf4ce44e0a76b1a0743305136" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 33457 } }
I20260810 06:22:39.239184 22078 leader_election.cc:288] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c4206f97a9f438ba0d23a82c999e439 (127.31.204.195:33429), 9841360cf4ce44e0a76b1a0743305136 (127.31.204.193:33457)
I20260810 06:22:39.245215 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:39.240335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:48871 (local address 127.31.204.193:33457)
0810 06:22:39.240501 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:39.240506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:39.240984 (+ 478us) server_negotiation.cc:407] Connection header received
0810 06:22:39.241129 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:39.241133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:39.241184 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:39.241278 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:22:39.241707 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.242443 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.243969 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.244529 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.244611 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:39.244664 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:39.244989 (+ 325us) server_negotiation.cc:299] Negotiation successful
0810 06:22:39.245068 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 06:22:39.245486 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:39.240726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:50439 (local address 127.31.204.195:33429)
0810 06:22:39.240842 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:39.240846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:39.241017 (+ 171us) server_negotiation.cc:407] Connection header received
0810 06:22:39.241739 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:39.241742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:39.241791 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:39.241854 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:22:39.242259 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.242941 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.244305 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.244860 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.244944 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:39.244988 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:39.245297 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 06:22:39.245377 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 06:22:39.245621 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:39.239585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:33457 (local address 127.31.204.194:48871)
0810 06:22:39.240886 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:39.240901 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:39.241026 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:39.241421 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:39.241425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:39.241448 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:22:39.241572 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:39.241577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.242561 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.242564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:39.243828 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:39.243839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.244755 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.244758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:39.244852 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:39.244875 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 06:22:39.245407 (+ 532us) client_negotiation.cc:240] Negotiation successful
0810 06:22:39.245478 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1227,"thread_start_us":141,"threads_started":1}
I20260810 06:22:39.246256 21486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35a453b171b4ae48c0e9584513aff36" candidate_uuid: "5e36a7b8113a4c53b9977bb9e91f9af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9841360cf4ce44e0a76b1a0743305136" is_pre_election: true
I20260810 06:22:39.246582 21486 raft_consensus.cc:2454] T a35a453b171b4ae48c0e9584513aff36 P 9841360cf4ce44e0a76b1a0743305136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e36a7b8113a4c53b9977bb9e91f9af4 in term 0.
I20260810 06:22:39.247102 21597 leader_election.cc:302] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e36a7b8113a4c53b9977bb9e91f9af4, 9841360cf4ce44e0a76b1a0743305136; no voters:
I20260810 06:22:39.247406 22078 raft_consensus.cc:2781] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:22:39.247498 22078 raft_consensus.cc:482] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:22:39.247548 22078 raft_consensus.cc:3037] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:22:39.248961 22078 raft_consensus.cc:504] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36a7b8113a4c53b9977bb9e91f9af4" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34161 } } peers { permanent_uuid: "9c4206f97a9f438ba0d23a82c999e439" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 33429 } } peers { permanent_uuid: "9841360cf4ce44e0a76b1a0743305136" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 33457 } }
I20260810 06:22:39.249323 22078 leader_election.cc:288] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [CANDIDATE]: Term 1 election: Requested vote from peers 9c4206f97a9f438ba0d23a82c999e439 (127.31.204.195:33429), 9841360cf4ce44e0a76b1a0743305136 (127.31.204.193:33457)
I20260810 06:22:39.249820 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:39.240593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:33429 (local address 127.31.204.194:50439)
0810 06:22:39.240937 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:39.240947 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:39.241063 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:39.241957 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:39.241961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:39.241979 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:22:39.242132 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:39.242138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.243078 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.243081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:39.244183 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:39.244192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.245068 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.245071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:39.245163 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:39.245183 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 06:22:39.245677 (+ 494us) client_negotiation.cc:240] Negotiation successful
0810 06:22:39.245863 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":8,"spinlock_wait_cycles":18816,"thread_start_us":91,"threads_started":1}
I20260810 06:22:39.249825 21486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35a453b171b4ae48c0e9584513aff36" candidate_uuid: "5e36a7b8113a4c53b9977bb9e91f9af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9841360cf4ce44e0a76b1a0743305136"
I20260810 06:22:39.249934 21486 raft_consensus.cc:3037] T a35a453b171b4ae48c0e9584513aff36 P 9841360cf4ce44e0a76b1a0743305136 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:22:39.250443 21942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35a453b171b4ae48c0e9584513aff36" candidate_uuid: "5e36a7b8113a4c53b9977bb9e91f9af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c4206f97a9f438ba0d23a82c999e439" is_pre_election: true
I20260810 06:22:39.250727 21942 raft_consensus.cc:2454] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e36a7b8113a4c53b9977bb9e91f9af4 in term 0.
I20260810 06:22:39.250820 21940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35a453b171b4ae48c0e9584513aff36" candidate_uuid: "5e36a7b8113a4c53b9977bb9e91f9af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c4206f97a9f438ba0d23a82c999e439"
I20260810 06:22:39.250959 21940 raft_consensus.cc:3037] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:22:39.251302 21486 raft_consensus.cc:2454] T a35a453b171b4ae48c0e9584513aff36 P 9841360cf4ce44e0a76b1a0743305136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e36a7b8113a4c53b9977bb9e91f9af4 in term 1.
I20260810 06:22:39.251605 21597 leader_election.cc:302] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e36a7b8113a4c53b9977bb9e91f9af4, 9841360cf4ce44e0a76b1a0743305136; no voters:
I20260810 06:22:39.251771 22078 raft_consensus.cc:2781] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:22:39.251971 22078 raft_consensus.cc:686] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [term 1 LEADER]: Becoming Leader. State: Replica: 5e36a7b8113a4c53b9977bb9e91f9af4, State: Running, Role: LEADER
I20260810 06:22:39.252215 21940 raft_consensus.cc:2454] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e36a7b8113a4c53b9977bb9e91f9af4 in term 1.
I20260810 06:22:39.252328 22078 consensus_queue.cc:237] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36a7b8113a4c53b9977bb9e91f9af4" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34161 } } peers { permanent_uuid: "9c4206f97a9f438ba0d23a82c999e439" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 33429 } } peers { permanent_uuid: "9841360cf4ce44e0a76b1a0743305136" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 33457 } }
I20260810 06:22:39.255551 21236 catalog_manager.cc:5495] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e36a7b8113a4c53b9977bb9e91f9af4 (127.31.204.194). New cstate: current_term: 1 leader_uuid: "5e36a7b8113a4c53b9977bb9e91f9af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36a7b8113a4c53b9977bb9e91f9af4" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 34161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c4206f97a9f438ba0d23a82c999e439" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 33429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9841360cf4ce44e0a76b1a0743305136" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 33457 } health_report { overall_health: UNKNOWN } } }
W20260810 06:22:39.256371 22018 tablet.cc:2369] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 06:22:39 dist-test-slave-q9h9 krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342958, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for HTTP/127.31.204.212@KRBTEST.COM
I20260810 06:22:39.275542 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:39.269090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56996 (local address 127.31.204.194:34161)
0810 06:22:39.269198 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:39.269202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:39.269214 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:22:39.269274 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:39.269277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:39.269331 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:39.269423 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:22:39.269813 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.270618 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.273095 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:39.273436 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:39.273536 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:39.274227 (+ 691us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:22:39.274756 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:22:39.274885 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:39.275340 (+ 455us) server_negotiation.cc:299] Negotiation successful
0810 06:22:39.275397 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:22:39.282325 21486 raft_consensus.cc:1261] T a35a453b171b4ae48c0e9584513aff36 P 9841360cf4ce44e0a76b1a0743305136 [term 1 FOLLOWER]: Refusing update from remote peer 5e36a7b8113a4c53b9977bb9e91f9af4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:22:39.282325 21940 raft_consensus.cc:1261] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439 [term 1 FOLLOWER]: Refusing update from remote peer 5e36a7b8113a4c53b9977bb9e91f9af4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:22:39.283003 22078 consensus_queue.cc:1035] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9841360cf4ce44e0a76b1a0743305136" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:22:39.283371 22078 consensus_queue.cc:1035] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c4206f97a9f438ba0d23a82c999e439" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 33429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:22:39.292780 22089 mvcc.cc:205] Tried to move back new op lower bound from 7316860761213640704 to 7316860761100312576. Current Snapshot: MvccSnapshot[applied={T|T < 7316860761213640704}]
I20260810 06:22:39.296021 22091 mvcc.cc:205] Tried to move back new op lower bound from 7316860761213640704 to 7316860761100312576. Current Snapshot: MvccSnapshot[applied={T|T < 7316860761213640704}]
I20260810 06:22:39.303735 21235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:22:39.303881 21235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:22:39.304168 22092 mvcc.cc:205] Tried to move back new op lower bound from 7316860761213640704 to 7316860761100312576. Current Snapshot: MvccSnapshot[applied={T|T < 7316860761213640704}]
I20260810 06:22:39.306275 21235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 893c17b1e4c2419b824b0410f24f9888: Sending DeleteTablet for 3 replicas of tablet a35a453b171b4ae48c0e9584513aff36
I20260810 06:22:39.307163 21681 tablet_service.cc:1514] Processing DeleteTablet for tablet a35a453b171b4ae48c0e9584513aff36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:22:39 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:56994
I20260810 06:22:39.307375 21882 tablet_service.cc:1514] Processing DeleteTablet for tablet a35a453b171b4ae48c0e9584513aff36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:22:39 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:46080
I20260810 06:22:39.307588 22103 tablet_replica.cc:330] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4: stopping tablet replica
I20260810 06:22:39.307869 22103 raft_consensus.cc:2229] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:22:39.307950 21445 tablet_service.cc:1514] Processing DeleteTablet for tablet a35a453b171b4ae48c0e9584513aff36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:22:39 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:58582
I20260810 06:22:39.309306 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 21288
I20260810 06:22:39.325917 22104 tablet_replica.cc:330] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439: stopping tablet replica
I20260810 06:22:39.326313 22103 raft_consensus.cc:2258] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:22:39.326375 22104 raft_consensus.cc:2229] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:22:39.328102 22103 ts_tablet_manager.cc:1905] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:22:39.330580 22103 ts_tablet_manager.cc:1918] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:22:39.330678 22103 log.cc:1196] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/a35a453b171b4ae48c0e9584513aff36
I20260810 06:22:39.330982 22103 ts_tablet_manager.cc:1939] T a35a453b171b4ae48c0e9584513aff36 P 5e36a7b8113a4c53b9977bb9e91f9af4: Deleting consensus metadata
I20260810 06:22:39.332072 21215 catalog_manager.cc:4841] TS 5e36a7b8113a4c53b9977bb9e91f9af4 (127.31.204.194:34161): tablet a35a453b171b4ae48c0e9584513aff36 (table test-table [id=e6fac965444d4c74bc2710741586cc28]) successfully deleted
I20260810 06:22:39.332197 22104 raft_consensus.cc:2258] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 06:22:39.332713 21214 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.204.193:33457 (error 108)
I20260810 06:22:39.333582 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 21563
I20260810 06:22:39.339692 22052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:39.338242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:33457 (local address 127.0.0.1:58586)
0810 06:22:39.339507 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:39.339593 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:33457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1170}
W20260810 06:22:39.340127 21214 catalog_manager.cc:4584] TS 9841360cf4ce44e0a76b1a0743305136 (127.31.204.193:33457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a35a453b171b4ae48c0e9584513aff36: Network error: Client connection negotiation failed: client connection to 127.31.204.193:33457: connect: Connection refused (error 111)
I20260810 06:22:39.340730 22104 ts_tablet_manager.cc:1905] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:22:39.343379 22104 ts_tablet_manager.cc:1918] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:22:39.343528 22104 log.cc:1196] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/a35a453b171b4ae48c0e9584513aff36
I20260810 06:22:39.343911 22104 ts_tablet_manager.cc:1939] T a35a453b171b4ae48c0e9584513aff36 P 9c4206f97a9f438ba0d23a82c999e439: Deleting consensus metadata
I20260810 06:22:39.345124 21212 catalog_manager.cc:4841] TS 9c4206f97a9f438ba0d23a82c999e439 (127.31.204.195:33429): tablet a35a453b171b4ae48c0e9584513aff36 (table test-table [id=e6fac965444d4c74bc2710741586cc28]) successfully deleted
I20260810 06:22:39.345300 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 21787
I20260810 06:22:39.362267 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 21188
I20260810 06:22:39.376971 32563 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 06:22:39.720614 32563 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 06:22:39.720722 32563 mini_ranger.cc:66] Stopping Ranger...
I20260810 06:22:40.064090 32563 mini_ranger.cc:68] Stopped Ranger
2026-08-10 06:22:40.064 UTC [19624] LOG: received smart shutdown request
2026-08-10 06:22:40.068 UTC [19624] LOG: background worker "logical replication launcher" (PID 19643) exited with exit code 1
2026-08-10 06:22:40.068 UTC [19631] LOG: shutting down
2026-08-10 06:22:40.193 UTC [19624] LOG: database system is shut down
2026-08-10T06:22:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45349 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:22:40 dist-test-slave-q9h9 krb5kdc[22127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:22:40 dist-test-slave-q9h9 krb5kdc[22127](info): set up 2 sockets
Aug 10 06:22:40 dist-test-slave-q9h9 krb5kdc[22127](info): commencing operation
krb5kdc: starting...
W20260810 06:22:41.439283 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:22:41 dist-test-slave-q9h9 krb5kdc[22127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:22:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:22:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:22:41.549834 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:43269
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:46645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:43269,127.31.204.253:45509,127.31.204.252:43783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:22:41.661554 22152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:41.661788 22152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:41.661825 22152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:41.665630 22152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:22:41.665678 22152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:41.665695 22152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:41.665710 22152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:22:41.665723 22152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:22:41.670671 22152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:46645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:43269,127.31.204.253:45509,127.31.204.252:43783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:43269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:41.671589 22152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:41.672516 22152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:22:41.680405 22160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:41.680400 22161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:41.680400 22163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:41.680586 22152 server_base.cc:1034] running on GCE node
I20260810 06:22:41.681154 22152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:41.682185 22152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:41.683350 22152 hybrid_clock.cc:648] HybridClock initialized: now 1786342961683329 us; error 34 us; skew 500 ppm
Aug 10 06:22:41 dist-test-slave-q9h9 krb5kdc[22127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342961, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:41.685876 22152 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:22:41.686745 22152 webserver.cc:460] Webserver started at http://127.31.204.254:45671/ using document root <none> and password file <none>
I20260810 06:22:41.687250 22152 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:41.687314 22152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:41.687498 22152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:22:41.689421 22152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "f84c14cede214dfe9234bcc31dd0dda5"
format_stamp: "Formatted at 2026-08-10 06:22:41 on dist-test-slave-q9h9"
server_key: "0a76e12560fd7db5089e3d7b9721c2d1"
server_key_iv: "e0963cfd9d8a9c53b217a725df0e9117"
server_key_version: "encryptionkey@0"
I20260810 06:22:41.689946 22152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "f84c14cede214dfe9234bcc31dd0dda5"
format_stamp: "Formatted at 2026-08-10 06:22:41 on dist-test-slave-q9h9"
server_key: "0a76e12560fd7db5089e3d7b9721c2d1"
server_key_iv: "e0963cfd9d8a9c53b217a725df0e9117"
server_key_version: "encryptionkey@0"
I20260810 06:22:41.693409 22152 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 06:22:41.695667 22178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:41.696760 22152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:22:41.696870 22152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "f84c14cede214dfe9234bcc31dd0dda5"
format_stamp: "Formatted at 2026-08-10 06:22:41 on dist-test-slave-q9h9"
server_key: "0a76e12560fd7db5089e3d7b9721c2d1"
server_key_iv: "e0963cfd9d8a9c53b217a725df0e9117"
server_key_version: "encryptionkey@0"
I20260810 06:22:41.696959 22152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:41.723923 22152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:41.726984 22152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:41.727169 22152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:41.733873 22152 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:43269
I20260810 06:22:41.734169 22287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:43269 every 8 connection(s)
I20260810 06:22:41.734800 22152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:22:41.735466 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 22152
I20260810 06:22:41.735601 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:22:41.735894 32563 external_mini_cluster.cc:1428] Setting key 205ccb0f4ad7579f22b41751bd0be8fb
I20260810 06:22:41.738263 22289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:22:41.741984 22289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.204.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.253@KRBTEST.COM" created.
Aug 10 06:22:41 dist-test-slave-q9h9 krb5kdc[22127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342961, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:22:41.757304 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:41.744268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.31.204.254:43269)
0810 06:22:41.744650 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:41.744664 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:41.744925 (+ 261us) server_negotiation.cc:407] Connection header received
0810 06:22:41.745947 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:41.745952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:41.746172 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:41.746343 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:41.747170 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:41.748310 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:41.749275 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:41.749918 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:41.750132 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:41.752592 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:41.752600 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:41.752614 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:41.752641 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:41.754420 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:41.754857 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:41.754862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:41.754863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:41.754976 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:41.755277 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:41.755281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:41.755282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:41.755635 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:41.755751 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:41.756098 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 06:22:41.756306 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":106,"threads_started":1}
I20260810 06:22:41.757304 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:41.744268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:43269 (local address 127.0.0.1:55546)
0810 06:22:41.744771 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:41.744821 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:41.745607 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:41.746348 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:41.746353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:41.746764 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:41.747012 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:41.747025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:41.748438 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:41.748440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:41.749155 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:41.749164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:41.750139 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:41.750143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:41.750238 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:41.750753 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:41.750771 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:41.752410 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:41.754526 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:41.754534 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:41.754541 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:41.754752 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:41.755080 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:41.755083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:41.755085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:41.755193 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:41.755744 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:41.755750 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:41.755876 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 06:22:41.756017 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 06:22:41.756395 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":87,"spinlock_wait_cycles":17280,"thread_start_us":139,"threads_started":1}
I20260810 06:22:41.759552 22289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.253.keytab.
Entry for principal kudu/127.31.204.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.253.keytab.
I20260810 06:22:41.760223 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:41.759953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:45509 (local address 127.0.0.1:49374)
0810 06:22:41.760088 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:41.760164 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.253:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 06:22:41.760398 22182 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.204.253:45509: connect: Connection refused (error 111)
I20260810 06:22:41.760964 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.253:45509
--webserver_interface=127.31.204.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.253.keytab
--principal=kudu/127.31.204.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:46645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:43269,127.31.204.253:45509,127.31.204.252:43783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:22:41.761210 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:41.760978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:45509 (local address 127.0.0.1:49380)
0810 06:22:41.761096 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:41.761151 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.253:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 06:22:41.761386 22289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.253:45509: Network error: Client connection negotiation failed: client connection to 127.31.204.253:45509: connect: Connection refused (error 111)
I20260810 06:22:41.810600 22289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } attempt: 1
I20260810 06:22:41.811487 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:41.811174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:45509 (local address 127.0.0.1:49392)
0810 06:22:41.811336 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:41.811404 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.253:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 06:22:41.812254 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:41.812040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:45509 (local address 127.0.0.1:49394)
0810 06:22:41.812136 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:41.812187 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.253:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 06:22:41.812386 22289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.253:45509: Network error: Client connection negotiation failed: client connection to 127.31.204.253:45509: connect: Connection refused (error 111)
W20260810 06:22:41.868778 22302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:41.869004 22302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:41.869041 22302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:41.872876 22302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:22:41.872925 22302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:41.872943 22302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:41.872958 22302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:22:41.872972 22302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:22:41.877948 22302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:46645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:43269,127.31.204.253:45509,127.31.204.252:43783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.253.keytab
--principal=kudu/127.31.204.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.253:45509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.204.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:41.878865 22302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:41.879794 22302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:22:41.880529 22289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } attempt: 2
I20260810 06:22:41.881313 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:41.881028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:45509 (local address 127.0.0.1:49400)
0810 06:22:41.881175 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:41.881234 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.253:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 06:22:41.881974 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:41.881792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:45509 (local address 127.0.0.1:49416)
0810 06:22:41.881875 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:41.881918 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.253:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 06:22:41.882140 22289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.253:45509: Network error: Client connection negotiation failed: client connection to 127.31.204.253:45509: connect: Connection refused (error 111)
W20260810 06:22:41.888640 22319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:41.888703 22315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:41.888703 22314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:41.888857 22302 server_base.cc:1034] running on GCE node
I20260810 06:22:41.889365 22302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:41.890370 22302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:41.891510 22302 hybrid_clock.cc:648] HybridClock initialized: now 1786342961891482 us; error 44 us; skew 500 ppm
Aug 10 06:22:41 dist-test-slave-q9h9 krb5kdc[22127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342961, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:41.894069 22302 init.cc:377] Logged in from keytab as kudu/127.31.204.253@KRBTEST.COM (short username kudu)
I20260810 06:22:41.894899 22302 webserver.cc:460] Webserver started at http://127.31.204.253:38181/ using document root <none> and password file <none>
I20260810 06:22:41.895401 22302 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:41.895468 22302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:41.895661 22302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:22:41.897536 22302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-1/data/instance:
uuid: "4662e94c8f90433d83c0c14b83187521"
format_stamp: "Formatted at 2026-08-10 06:22:41 on dist-test-slave-q9h9"
server_key: "d57d2e0be33eb9c61166f106391e2a4a"
server_key_iv: "ea4f2c6931032397739032c689546923"
server_key_version: "encryptionkey@0"
I20260810 06:22:41.898087 22302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-1/wal/instance:
uuid: "4662e94c8f90433d83c0c14b83187521"
format_stamp: "Formatted at 2026-08-10 06:22:41 on dist-test-slave-q9h9"
server_key: "d57d2e0be33eb9c61166f106391e2a4a"
server_key_iv: "ea4f2c6931032397739032c689546923"
server_key_version: "encryptionkey@0"
I20260810 06:22:41.901507 22302 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:22:41.903719 22334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:41.904795 22302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:22:41.904897 22302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-1/wal
uuid: "4662e94c8f90433d83c0c14b83187521"
format_stamp: "Formatted at 2026-08-10 06:22:41 on dist-test-slave-q9h9"
server_key: "d57d2e0be33eb9c61166f106391e2a4a"
server_key_iv: "ea4f2c6931032397739032c689546923"
server_key_version: "encryptionkey@0"
I20260810 06:22:41.904987 22302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:41.915845 22302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:41.918274 22302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:41.918426 22302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:41.924572 22302 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.253:45509
I20260810 06:22:41.924799 22417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.253:45509 every 8 connection(s)
I20260810 06:22:41.925411 22302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-1/data/info.pb
I20260810 06:22:41.926493 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 22302
I20260810 06:22:41.926613 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-1/wal/instance
I20260810 06:22:41.926859 32563 external_mini_cluster.cc:1428] Setting key ff570421c91493ec3b4cdb2c13340060
I20260810 06:22:41.927980 22418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:22:41.931380 22418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.204.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.252@KRBTEST.COM" created.
Aug 10 06:22:41 dist-test-slave-q9h9 krb5kdc[22127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342961, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.253@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:22:41.943811 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:41.932990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55556 (local address 127.31.204.254:43269)
0810 06:22:41.933144 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:41.933149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:41.933852 (+ 703us) server_negotiation.cc:407] Connection header received
0810 06:22:41.934720 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:41.934724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:41.934788 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:41.934885 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:41.935909 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:41.936430 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:41.937714 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:41.937944 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:41.938055 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:41.940428 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:41.940432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:41.940434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:41.940463 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:41.941926 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:41.942408 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:41.942412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:41.942413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:41.942501 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:41.942780 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:41.942783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:41.942785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:41.943000 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:41.943110 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:41.943525 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 06:22:41.943646 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:22:41.944350 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:41.933275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:43269 (local address 127.0.0.1:55556)
0810 06:22:41.933690 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:41.933738 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:41.934521 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:41.935021 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:41.935027 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:41.935317 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:41.935774 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:41.935787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:41.936524 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:41.936527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:41.937595 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:41.937608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:41.938111 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:41.938115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:41.938310 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:41.938825 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:41.938841 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:41.940253 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:41.942057 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:41.942064 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:41.942074 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:41.942316 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:41.942581 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:41.942583 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:41.942585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:41.942694 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:41.943085 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:41.943089 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:41.943289 (+ 200us) client_negotiation.cc:769] Sending connection context
0810 06:22:41.943418 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 06:22:41.943624 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":97,"threads_started":1}
I20260810 06:22:41.945538 22418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.252.keytab.
Entry for principal kudu/127.31.204.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.252.keytab.
Aug 10 06:22:41 dist-test-slave-q9h9 krb5kdc[22127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342961, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.253@KRBTEST.COM for kudu/127.31.204.253@KRBTEST.COM
I20260810 06:22:41.950392 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.252:43783
--webserver_interface=127.31.204.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.252.keytab
--principal=kudu/127.31.204.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:46645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:43269,127.31.204.253:45509,127.31.204.252:43783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:22:41.953473 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:41.945922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:45509 (local address 127.0.0.1:49428)
0810 06:22:41.946044 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:41.946058 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:41.946134 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:41.946590 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:41.946592 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:41.946722 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:41.946801 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:41.946806 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:41.947726 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:41.947729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:41.948232 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:41.948240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:41.948799 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:41.948802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:41.948871 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:41.949291 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:41.949304 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:41.950407 (+ 1103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:41.952163 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:41.952166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:41.952168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:41.952344 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:41.952589 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:41.952592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:41.952594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:41.952665 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:41.953042 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:41.953045 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:41.953123 (+ 78us) client_negotiation.cc:769] Sending connection context
0810 06:22:41.953208 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 06:22:41.953326 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 06:22:41.953577 22426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:41.946036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49428 (local address 127.31.204.253:45509)
0810 06:22:41.946277 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:41.946290 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:41.946307 (+ 17us) server_negotiation.cc:407] Connection header received
0810 06:22:41.946347 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:41.946350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:41.946476 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:41.946571 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:41.946890 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:41.947629 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:41.948326 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:41.948685 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:41.948786 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:41.950554 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:41.950564 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:41.950576 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:41.950601 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:41.952068 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:41.952434 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:41.952440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:41.952441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:41.952513 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:41.952735 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:41.952739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:41.952740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:41.952949 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:41.953057 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:41.953290 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 06:22:41.953403 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":77,"threads_started":1}
I20260810 06:22:41.955210 22418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:22:41.955821 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:41.955550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43783 (local address 127.0.0.1:40172)
0810 06:22:41.955671 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:41.955751 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 06:22:41.956010 22340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43783: connect: Connection refused (error 111)
I20260810 06:22:41.956771 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:41.956541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43783 (local address 127.0.0.1:40186)
0810 06:22:41.956656 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:41.956708 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 06:22:41.956919 22418 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.252:43783: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43783: connect: Connection refused (error 111)
I20260810 06:22:41.973289 22289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } attempt: 3
Aug 10 06:22:41 dist-test-slave-q9h9 krb5kdc[22127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342961, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for kudu/127.31.204.253@KRBTEST.COM
I20260810 06:22:41.981017 22426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:41.973758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49442 (local address 127.31.204.253:45509)
0810 06:22:41.973862 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:41.973866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:41.973878 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:22:41.973959 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:41.973962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:41.974000 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:41.974088 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:41.974462 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:41.974949 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:41.976123 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:41.976349 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:41.976433 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:41.978443 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:41.978447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:41.978450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:41.978473 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:41.979664 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:41.980095 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:41.980097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:41.980098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:41.980157 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:41.980396 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:41.980397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:41.980399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:41.980532 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:41.980595 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:41.980846 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 06:22:41.980925 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:22:41.981086 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:41.973678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:45509 (local address 127.0.0.1:49442)
0810 06:22:41.973786 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:41.973800 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:41.973880 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:41.974105 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:41.974109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:41.974258 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:41.974361 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:41.974366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:41.975043 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:41.975046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:41.976012 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:41.976022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:41.976446 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:41.976449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:41.976544 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:41.976947 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:41.976957 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:41.978285 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:41.979764 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:41.979768 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:41.979770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:41.980011 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:41.980229 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:41.980232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:41.980234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:41.980323 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:41.980605 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:41.980608 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:41.980713 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 06:22:41.980818 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 06:22:41.980942 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 06:22:41.981993 22289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:22:41.982630 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:41.982394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43783 (local address 127.0.0.1:40196)
0810 06:22:41.982506 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:41.982562 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 06:22:41.983327 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:41.983087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43783 (local address 127.0.0.1:40202)
0810 06:22:41.983217 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:41.983267 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260810 06:22:41.983453 22289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.252:43783: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43783: connect: Connection refused (error 111)
I20260810 06:22:42.006119 22418 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } attempt: 1
I20260810 06:22:42.006688 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.006415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43783 (local address 127.0.0.1:40208)
0810 06:22:42.006541 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:42.006613 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 06:22:42.007305 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.007083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43783 (local address 127.0.0.1:40216)
0810 06:22:42.007185 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:42.007241 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 06:22:42.007438 22418 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.252:43783: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43783: connect: Connection refused (error 111)
I20260810 06:22:42.014590 22289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } attempt: 1
I20260810 06:22:42.015169 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.014917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43783 (local address 127.0.0.1:40220)
0810 06:22:42.015034 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:42.015095 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 06:22:42.015817 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.015606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43783 (local address 127.0.0.1:40232)
0810 06:22:42.015698 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:42.015752 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 06:22:42.015946 22289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.252:43783: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43783: connect: Connection refused (error 111)
W20260810 06:22:42.060637 22428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:42.060849 22428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:42.060899 22428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:42.064774 22428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:22:42.064826 22428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:42.064846 22428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:42.064881 22428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:22:42.064913 22428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:22:42.069844 22428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:46645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:43269,127.31.204.253:45509,127.31.204.252:43783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.252.keytab
--principal=kudu/127.31.204.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.252:43783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.204.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:42.070751 22428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:42.071653 22428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:22:42.075572 22418 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } attempt: 2
I20260810 06:22:42.076344 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.076074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43783 (local address 127.0.0.1:40242)
0810 06:22:42.076207 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:42.076272 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 06:22:42.077047 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.076843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43783 (local address 127.0.0.1:40244)
0810 06:22:42.076929 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:42.076975 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 06:22:42.077201 22418 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.252:43783: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43783: connect: Connection refused (error 111)
I20260810 06:22:42.079752 22428 server_base.cc:1034] running on GCE node
W20260810 06:22:42.079805 22441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:42.079736 22444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:42.079921 22442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:42.080399 22428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:42.081321 22428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:42.082468 22428 hybrid_clock.cc:648] HybridClock initialized: now 1786342962082460 us; error 25 us; skew 500 ppm
Aug 10 06:22:42 dist-test-slave-q9h9 krb5kdc[22127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342962, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:42.084971 22428 init.cc:377] Logged in from keytab as kudu/127.31.204.252@KRBTEST.COM (short username kudu)
I20260810 06:22:42.085753 22428 webserver.cc:460] Webserver started at http://127.31.204.252:38311/ using document root <none> and password file <none>
I20260810 06:22:42.086292 22428 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:42.086356 22428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:42.086541 22428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:22:42.088370 22428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-2/data/instance:
uuid: "6a725ccb270d476da22dceeec315bfe2"
format_stamp: "Formatted at 2026-08-10 06:22:42 on dist-test-slave-q9h9"
server_key: "c6c82ee19f29c6fc292e4505f0788e3e"
server_key_iv: "de9e9da382a54e8c0324765f65944b89"
server_key_version: "encryptionkey@0"
I20260810 06:22:42.088887 22428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-2/wal/instance:
uuid: "6a725ccb270d476da22dceeec315bfe2"
format_stamp: "Formatted at 2026-08-10 06:22:42 on dist-test-slave-q9h9"
server_key: "c6c82ee19f29c6fc292e4505f0788e3e"
server_key_iv: "de9e9da382a54e8c0324765f65944b89"
server_key_version: "encryptionkey@0"
I20260810 06:22:42.091089 22289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } attempt: 2
I20260810 06:22:42.091713 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.091422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43783 (local address 127.0.0.1:40258)
0810 06:22:42.091567 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:42.091630 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 06:22:42.092348 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.092168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43783 (local address 127.0.0.1:40264)
0810 06:22:42.092251 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:42.092294 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 06:22:42.092461 22289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.252:43783: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43783: connect: Connection refused (error 111)
I20260810 06:22:42.092460 22428 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260810 06:22:42.094609 22459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:42.095696 22428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 06:22:42.095813 22428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-2/wal
uuid: "6a725ccb270d476da22dceeec315bfe2"
format_stamp: "Formatted at 2026-08-10 06:22:42 on dist-test-slave-q9h9"
server_key: "c6c82ee19f29c6fc292e4505f0788e3e"
server_key_iv: "de9e9da382a54e8c0324765f65944b89"
server_key_version: "encryptionkey@0"
I20260810 06:22:42.095898 22428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:42.116567 22428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:42.119289 22428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:42.119447 22428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:42.125574 22428 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.252:43783
I20260810 06:22:42.125834 22542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.252:43783 every 8 connection(s)
I20260810 06:22:42.126464 22428 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-2/data/info.pb
I20260810 06:22:42.129029 22545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:22:42.132390 22545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:22:42.135960 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 22428
I20260810 06:22:42.136080 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786342803670820-32563-0/minicluster-data/master-2/wal/instance
I20260810 06:22:42.136389 32563 external_mini_cluster.cc:1428] Setting key ece204cbb503ecd603046f2fda52a414
Aug 10 06:22:42 dist-test-slave-q9h9 krb5kdc[22127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342962, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.252@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
Aug 10 06:22:42 dist-test-slave-q9h9 krb5kdc[22127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.252@KRBTEST.COM
I20260810 06:22:42.144883 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.133993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.31.204.254:43269)
0810 06:22:42.134141 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:42.134145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:42.134956 (+ 811us) server_negotiation.cc:407] Connection header received
0810 06:22:42.135742 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:42.135745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:42.135791 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:42.135858 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:42.137195 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.137877 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.138951 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.139169 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.139256 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:42.141486 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:42.141489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:42.141492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:42.141518 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:42.142879 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.143517 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.143521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.143522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.143614 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.143947 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.143949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.143950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.144120 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:42.144188 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:42.144667 (+ 479us) server_negotiation.cc:299] Negotiation successful
0810 06:22:42.144772 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 06:22:42.145803 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.134387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:43269 (local address 127.0.0.1:55566)
0810 06:22:42.134819 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:42.134858 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:42.135558 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:42.136043 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:42.136051 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:42.136466 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:42.137031 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:42.137045 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.137988 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.137991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:42.138835 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:42.138845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.139307 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.139311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:42.139460 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:42.139937 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:42.139953 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:42.141350 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:42.142977 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:42.142986 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:42.142995 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:42.143427 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:42.143699 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:42.143703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:42.143705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:42.143867 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:42.144219 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:42.144225 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:42.144467 (+ 242us) client_negotiation.cc:769] Sending connection context
0810 06:22:42.144630 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 06:22:42.144910 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":115,"threads_started":1}
I20260810 06:22:42.145934 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.137014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.31.204.252:43783)
0810 06:22:42.137381 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:42.137389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:42.137409 (+ 20us) server_negotiation.cc:407] Connection header received
0810 06:22:42.137477 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:42.137481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:42.137613 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:42.137756 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:42.138493 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.139219 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.140005 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.140338 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.140429 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:42.142614 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:42.142618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:42.142627 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:42.142648 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:42.144220 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.144581 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.144586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.144588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.144672 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.144992 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.144995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.144996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.145249 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:42.145425 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:42.145609 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 06:22:42.145747 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":139,"threads_started":1}
I20260810 06:22:42.146930 22545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:22:42 dist-test-slave-q9h9 krb5kdc[22127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342962, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.252@KRBTEST.COM for kudu/127.31.204.253@KRBTEST.COM
Aug 10 06:22:42 dist-test-slave-q9h9 krb5kdc[22127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:22:42.154358 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.147312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:45509 (local address 127.0.0.1:49452)
0810 06:22:42.147409 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:42.147422 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:42.147481 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:42.147758 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:42.147761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:42.147916 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:42.148023 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:42.148030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.148701 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.148705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:42.149175 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:42.149182 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.149588 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.149591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:42.149686 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:42.150106 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:42.150120 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:42.151449 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:42.153043 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:42.153046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:42.153048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:42.153257 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:42.153525 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:42.153528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:42.153530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:42.153602 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:42.153981 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:42.153982 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:42.154095 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 06:22:42.154171 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 06:22:42.154254 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 06:22:42.154506 22426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.147433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49452 (local address 127.31.204.253:45509)
0810 06:22:42.147546 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:42.147550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:42.147564 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:22:42.147605 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:42.147607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:42.147652 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:42.147724 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:42.148134 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.148605 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.149288 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.149497 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.149580 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:42.151577 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:42.151580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:42.151582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:42.151603 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:42.152947 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.153355 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.153358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.153360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.153436 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.153680 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.153683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.153685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.153896 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:42.154000 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:42.154247 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 06:22:42.154356 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 06:22:42.155195 22545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:22:42.155803 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.148015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55580 (local address 127.31.204.254:43269)
0810 06:22:42.148108 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:42.148112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:42.148126 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:22:42.148234 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:42.148237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:42.148284 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:42.148343 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:42.148855 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.149298 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.150066 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.150391 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.150499 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:42.152718 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:42.152722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:42.152724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:42.152750 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:42.154138 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.154578 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.154582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.154584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.154672 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.154951 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.154954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.154956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.155161 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:42.155363 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:42.155476 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 06:22:42.155573 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 06:22:42 dist-test-slave-q9h9 krb5kdc[22127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342962, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.252@KRBTEST.COM for kudu/127.31.204.252@KRBTEST.COM
Aug 10 06:22:42 dist-test-slave-q9h9 krb5kdc[22127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.253@KRBTEST.COM
I20260810 06:22:42.162796 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.155561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43783 (local address 127.0.0.1:40278)
0810 06:22:42.155703 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:42.155713 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:42.155770 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:42.156129 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:42.156132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:42.156271 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:42.156336 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:42.156341 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.157169 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.157172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:42.157707 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:42.157716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.158189 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.158192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:42.158274 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:42.158641 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:42.158654 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:42.159750 (+ 1096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:42.161351 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:42.161353 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:42.161354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:42.161527 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:42.161783 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:42.161786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:42.161788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:42.161875 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:42.162283 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:42.162286 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:42.162373 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 06:22:42.162476 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 06:22:42.162592 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260810 06:22:42.162796 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.155592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.31.204.252:43783)
0810 06:22:42.155961 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:42.155964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:42.155974 (+ 10us) server_negotiation.cc:407] Connection header received
0810 06:22:42.156000 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:42.156002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:42.156032 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:42.156089 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:42.156448 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.157071 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.157815 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.158096 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.158175 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:42.159867 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:42.159870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:42.159872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:42.159892 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:42.161249 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.161623 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.161626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.161627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.161701 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.161960 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.161963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.161965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.162176 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:42.162259 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:42.162543 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 06:22:42.162643 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":312}
I20260810 06:22:42.164652 22426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.156886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49458 (local address 127.31.204.253:45509)
0810 06:22:42.156978 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:42.156981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:42.156991 (+ 10us) server_negotiation.cc:407] Connection header received
0810 06:22:42.157078 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:42.157080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:42.157110 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:42.157181 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:42.157670 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.158220 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.158912 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.159179 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.159254 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:42.161611 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:42.161615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:42.161618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:42.161641 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:42.163182 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.163566 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.163570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.163571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.163655 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.163905 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.163908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.163910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.164061 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:42.164272 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:42.164398 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 06:22:42.164500 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 06:22:42.166082 22545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2: Bootstrap starting.
I20260810 06:22:42.167672 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:22:42.168362 22418 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } attempt: 3
I20260810 06:22:42.168593 22545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2: Neither blocks nor log segments found. Creating new log.
I20260810 06:22:42.169458 22545 log.cc:824] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2: Log is configured to *not* fsync() on all Append() calls
I20260810 06:22:42.172379 22545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2: No bootstrap required, opened a new log
Aug 10 06:22:42 dist-test-slave-q9h9 krb5kdc[22127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342961, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.253@KRBTEST.COM for kudu/127.31.204.252@KRBTEST.COM
I20260810 06:22:42.176374 22545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } } peers { permanent_uuid: "4662e94c8f90433d83c0c14b83187521" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } } peers { permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } }
I20260810 06:22:42.176738 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.168955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.31.204.252:43783)
0810 06:22:42.169045 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:42.169048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:42.169060 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:22:42.169091 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:42.169093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:42.169129 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:42.169208 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:42.169665 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.170333 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.171357 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.171649 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.171762 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:42.173367 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:42.173370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:42.173372 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:42.173400 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:42.174606 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.175054 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.175057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.175059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.175156 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.175449 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.175452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.175453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.175665 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:42.175745 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:42.176440 (+ 695us) server_negotiation.cc:299] Negotiation successful
0810 06:22:42.176544 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 06:22:42.176779 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.168734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43783 (local address 127.0.0.1:40284)
0810 06:22:42.168854 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:42.168869 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:42.168941 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:42.169242 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:42.169244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:42.169406 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:42.169494 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:42.169502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.170446 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.170449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:42.171215 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:42.171226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.171779 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.171782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:42.171841 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:42.172196 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:42.172208 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:42.173237 (+ 1029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:42.174706 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:42.174709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:42.174711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:42.174955 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:42.175253 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:42.175256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:42.175270 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:42.175355 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:42.175786 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:42.175789 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:42.175873 (+ 84us) client_negotiation.cc:769] Sending connection context
0810 06:22:42.175968 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 06:22:42.176088 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 06:22:42.176628 22545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:22:42.177083 22545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a725ccb270d476da22dceeec315bfe2, State: Initialized, Role: FOLLOWER
I20260810 06:22:42.177709 22545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } } peers { permanent_uuid: "4662e94c8f90433d83c0c14b83187521" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } } peers { permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } }
I20260810 06:22:42.178223 22567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } } peers { permanent_uuid: "4662e94c8f90433d83c0c14b83187521" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } } peers { permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } } }
I20260810 06:22:42.178390 22567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:22:42.179098 22545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:22:42.180096 22418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521: Bootstrap starting.
I20260810 06:22:42.182252 22426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.169779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49466 (local address 127.31.204.253:45509)
0810 06:22:42.170101 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:42.170105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:42.170128 (+ 23us) server_negotiation.cc:407] Connection header received
0810 06:22:42.170267 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:42.170271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:42.170308 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:42.170352 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:42.174991 (+ 4639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.175623 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.176769 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.177043 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.177110 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:42.178925 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:42.178929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:42.178931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:42.178956 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:42.180502 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.180941 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.180944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.180946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.181025 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.181318 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.181321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.181323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.181514 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:42.181638 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:42.181965 (+ 327us) server_negotiation.cc:299] Negotiation successful
0810 06:22:42.182122 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258}
I20260810 06:22:42.183001 22418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521: Neither blocks nor log segments found. Creating new log.
I20260810 06:22:42.183845 22418 log.cc:824] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521: Log is configured to *not* fsync() on all Append() calls
W20260810 06:22:42.184932 22582 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:22:42.185004 22582 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:22:42.187279 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.169898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.31.204.254:43269)
0810 06:22:42.170000 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:42.170005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:42.172207 (+ 2202us) server_negotiation.cc:407] Connection header received
0810 06:22:42.172379 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:42.172382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:42.172429 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:42.172479 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:42.173410 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.174165 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.175318 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.175645 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.175726 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:42.177341 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:42.177344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:42.177347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:42.177373 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:42.185410 (+ 8037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.185859 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.185862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.185864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.185958 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.186285 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.186289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.186291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.186508 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:42.186607 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:42.186989 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 06:22:42.187116 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:22:42.182839 22428 master_runner.cc:186] Error getting master registration for 127.31.204.254:43269: OK, instance_id { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" instance_seqno: 1786342961732050 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 06:22:42.187640 22428 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 06:22:42.190575 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.170289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.31.204.252:43783)
0810 06:22:42.171983 (+ 1694us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:42.171991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:42.179073 (+ 7082us) server_negotiation.cc:407] Connection header received
0810 06:22:42.179302 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:42.179305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:42.179350 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:42.179411 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:42.180466 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.181240 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.184497 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.184873 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.184946 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:42.186548 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:42.186552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:42.186554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:42.186580 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:42.188783 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.189145 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.189149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.189150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.189267 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.189493 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.189496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.189498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.189704 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:42.189836 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:42.189983 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 06:22:42.190395 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1635,"thread_start_us":78,"threads_started":1}
I20260810 06:22:42.191619 22289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } attempt: 3
I20260810 06:22:42.191752 22418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521: No bootstrap required, opened a new log
I20260810 06:22:42.195653 22418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } } peers { permanent_uuid: "4662e94c8f90433d83c0c14b83187521" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } } peers { permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } }
I20260810 06:22:42.195888 22418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:22:42.195946 22418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4662e94c8f90433d83c0c14b83187521, State: Initialized, Role: FOLLOWER
I20260810 06:22:42.196414 22418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } } peers { permanent_uuid: "4662e94c8f90433d83c0c14b83187521" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } } peers { permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } }
Aug 10 06:22:42 dist-test-slave-q9h9 krb5kdc[22127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342961, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for kudu/127.31.204.252@KRBTEST.COM
I20260810 06:22:42.197717 22418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:22:42.201684 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.191997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43783 (local address 127.0.0.1:40308)
0810 06:22:42.192159 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:42.192173 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:42.192252 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:42.192467 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:42.192470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:42.192633 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:42.192728 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:42.192735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.193719 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.193722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:42.194579 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:42.194590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.195175 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.195178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:42.195257 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:42.195645 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:42.195658 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:42.197113 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:42.200117 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:42.200120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:42.200123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:42.200374 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:42.200659 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:42.200662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:42.200664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:42.200759 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:42.201186 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:42.201190 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:42.201302 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 06:22:42.201407 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 06:22:42.201540 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260810 06:22:42.202528 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:42.192062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40308 (local address 127.31.204.252:43783)
0810 06:22:42.192160 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:42.192164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:42.192217 (+ 53us) server_negotiation.cc:407] Connection header received
0810 06:22:42.192321 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:42.192324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:42.192364 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:42.192442 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:42.192852 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.193607 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.194701 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:42.195055 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:42.195337 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:42.198320 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:42.198324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:42.198327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:42.198350 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:42.199942 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.200478 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.200481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.200483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.200573 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:42.200857 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:42.200860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:42.200861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:42.201067 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:42.201162 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:42.201481 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 06:22:42.202391 (+ 910us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 06:22:42.202008 22302 master_runner.cc:186] Error getting master registration for 127.31.204.254:43269: OK, instance_id { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" instance_seqno: 1786342961732050 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 06:22:42.205008 22302 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 06:22:42.205415 22289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5: Bootstrap starting.
I20260810 06:22:42.206171 22587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } } peers { permanent_uuid: "4662e94c8f90433d83c0c14b83187521" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } } peers { permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } } }
I20260810 06:22:42.206314 22587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:22:42.208158 22289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5: Neither blocks nor log segments found. Creating new log.
W20260810 06:22:42.208617 22600 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:22:42.208675 22600 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:22:42.209789 22289 log.cc:824] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5: Log is configured to *not* fsync() on all Append() calls
I20260810 06:22:42.211784 22289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5: No bootstrap required, opened a new log
I20260810 06:22:42.215652 22289 raft_consensus.cc:348] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } } peers { permanent_uuid: "4662e94c8f90433d83c0c14b83187521" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } } peers { permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } }
I20260810 06:22:42.215865 22289 raft_consensus.cc:374] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:22:42.215916 22289 raft_consensus.cc:729] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f84c14cede214dfe9234bcc31dd0dda5, State: Initialized, Role: FOLLOWER
I20260810 06:22:42.216408 22289 consensus_queue.cc:260] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } } peers { permanent_uuid: "4662e94c8f90433d83c0c14b83187521" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } } peers { permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } }
I20260810 06:22:42.217686 22289 sys_catalog.cc:564] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:22:42.218286 22603 sys_catalog.cc:455] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } } peers { permanent_uuid: "4662e94c8f90433d83c0c14b83187521" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } } peers { permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } } }
I20260810 06:22:42.218475 22603 sys_catalog.cc:458] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 06:22:42.222735 22621 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:22:42.222811 22621 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:22:42.228080 22152 master_runner.cc:263] No leader master found from master f84c14cede214dfe9234bcc31dd0dda5
I20260810 06:22:42.228142 22152 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 06:22:42.228643 22194 tablet.cc:2369] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:22:42.365972 22587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:22:42.366209 22587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } } peers { permanent_uuid: "4662e94c8f90433d83c0c14b83187521" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } } peers { permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } }
I20260810 06:22:42.367344 22587 leader_election.cc:288] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f84c14cede214dfe9234bcc31dd0dda5 (127.31.204.254:43269), 6a725ccb270d476da22dceeec315bfe2 (127.31.204.252:43783)
I20260810 06:22:42.368049 22229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4662e94c8f90433d83c0c14b83187521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f84c14cede214dfe9234bcc31dd0dda5" is_pre_election: true
I20260810 06:22:42.368391 22229 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4662e94c8f90433d83c0c14b83187521 in term 0.
I20260810 06:22:42.369175 22499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4662e94c8f90433d83c0c14b83187521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a725ccb270d476da22dceeec315bfe2" is_pre_election: true
I20260810 06:22:42.369458 22499 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4662e94c8f90433d83c0c14b83187521 in term 0.
I20260810 06:22:42.370137 22340 leader_election.cc:302] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4662e94c8f90433d83c0c14b83187521, 6a725ccb270d476da22dceeec315bfe2; no voters:
I20260810 06:22:42.370436 22587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:22:42.370504 22587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:22:42.370548 22587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [term 0 FOLLOWER]: Advancing to term 1
W20260810 06:22:42.370647 22467 tablet.cc:2369] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:22:42.371907 22587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } } peers { permanent_uuid: "4662e94c8f90433d83c0c14b83187521" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } } peers { permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } }
I20260810 06:22:42.372280 22587 leader_election.cc:288] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [CANDIDATE]: Term 1 election: Requested vote from peers f84c14cede214dfe9234bcc31dd0dda5 (127.31.204.254:43269), 6a725ccb270d476da22dceeec315bfe2 (127.31.204.252:43783)
I20260810 06:22:42.373034 22499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4662e94c8f90433d83c0c14b83187521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a725ccb270d476da22dceeec315bfe2"
I20260810 06:22:42.373162 22499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:22:42.374562 22499 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4662e94c8f90433d83c0c14b83187521 in term 1.
I20260810 06:22:42.374980 22340 leader_election.cc:302] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4662e94c8f90433d83c0c14b83187521, 6a725ccb270d476da22dceeec315bfe2; no voters:
I20260810 06:22:42.375212 22587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:22:42.375420 22587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [term 1 LEADER]: Becoming Leader. State: Replica: 4662e94c8f90433d83c0c14b83187521, State: Running, Role: LEADER
I20260810 06:22:42.375742 22587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } } peers { permanent_uuid: "4662e94c8f90433d83c0c14b83187521" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } } peers { permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } }
I20260810 06:22:42.377471 22587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4662e94c8f90433d83c0c14b83187521. Latest consensus state: current_term: 1 leader_uuid: "4662e94c8f90433d83c0c14b83187521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } } peers { permanent_uuid: "4662e94c8f90433d83c0c14b83187521" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } } peers { permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } } }
I20260810 06:22:42.377599 22587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [sys.catalog]: This master's current role is: LEADER
I20260810 06:22:42.378960 22229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4662e94c8f90433d83c0c14b83187521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f84c14cede214dfe9234bcc31dd0dda5"
I20260810 06:22:42.379115 22229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:22:42.380463 22229 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4662e94c8f90433d83c0c14b83187521 in term 1.
I20260810 06:22:42.384965 22631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:22:42.386400 22631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:22:42.393287 22499 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2 [term 1 FOLLOWER]: Refusing update from remote peer 4662e94c8f90433d83c0c14b83187521: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:22:42.394474 22229 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5 [term 1 FOLLOWER]: Refusing update from remote peer 4662e94c8f90433d83c0c14b83187521: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:22:42.395241 22624 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:22:42.395534 22587 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:22:42.400017 22567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4662e94c8f90433d83c0c14b83187521. Latest consensus state: current_term: 1 leader_uuid: "4662e94c8f90433d83c0c14b83187521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } } peers { permanent_uuid: "4662e94c8f90433d83c0c14b83187521" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } } peers { permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } } }
I20260810 06:22:42.400148 22567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:22:42.401012 22603 sys_catalog.cc:455] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4662e94c8f90433d83c0c14b83187521. Latest consensus state: current_term: 1 leader_uuid: "4662e94c8f90433d83c0c14b83187521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } } peers { permanent_uuid: "4662e94c8f90433d83c0c14b83187521" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } } peers { permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } } }
I20260810 06:22:42.401152 22603 sys_catalog.cc:458] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:22:42.411031 22603 sys_catalog.cc:455] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4662e94c8f90433d83c0c14b83187521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } } peers { permanent_uuid: "4662e94c8f90433d83c0c14b83187521" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } } peers { permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } } }
I20260810 06:22:42.411024 22587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4662e94c8f90433d83c0c14b83187521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } } peers { permanent_uuid: "4662e94c8f90433d83c0c14b83187521" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } } peers { permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } } }
I20260810 06:22:42.411141 22603 sys_catalog.cc:458] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:22:42.411087 22567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4662e94c8f90433d83c0c14b83187521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } } peers { permanent_uuid: "4662e94c8f90433d83c0c14b83187521" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } } peers { permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } } }
I20260810 06:22:42.411152 22587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [sys.catalog]: This master's current role is: LEADER
I20260810 06:22:42.411187 22567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:22:42.411500 22635 mvcc.cc:205] Tried to move back new op lower bound from 7316860773950808064 to 7316860773893816320. Current Snapshot: MvccSnapshot[applied={T|T < 7316860773950808064}]
I20260810 06:22:42.411455 22587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4662e94c8f90433d83c0c14b83187521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84c14cede214dfe9234bcc31dd0dda5" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 43269 } } peers { permanent_uuid: "4662e94c8f90433d83c0c14b83187521" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 45509 } } peers { permanent_uuid: "6a725ccb270d476da22dceeec315bfe2" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43783 } } }
I20260810 06:22:42.411582 22587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521 [sys.catalog]: This master's current role is: LEADER
I20260810 06:22:42.411664 22634 mvcc.cc:205] Tried to move back new op lower bound from 7316860773950808064 to 7316860773893816320. Current Snapshot: MvccSnapshot[applied={T|T < 7316860773950808064}]
I20260810 06:22:42.411468 22633 mvcc.cc:205] Tried to move back new op lower bound from 7316860773950808064 to 7316860773893816320. Current Snapshot: MvccSnapshot[applied={T|T < 7316860773950808064}]
I20260810 06:22:42.414528 22631 catalog_manager.cc:1347] Generated new cluster ID: 8953f45fec83456b878495a41d75c199
I20260810 06:22:42.414608 22631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:22:42.431269 22631 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:22:42.432578 22631 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:22:42.444173 22631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4662e94c8f90433d83c0c14b83187521: Generated new TSK 0
I20260810 06:22:42.444950 22631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:22:43.186424 22582 catalog_manager.cc:1259] Loaded cluster ID: 8953f45fec83456b878495a41d75c199
I20260810 06:22:43.186506 22582 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2: loading cluster ID for follower catalog manager: success
I20260810 06:22:43.188189 22582 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2: acquiring CA information for follower catalog manager: success
I20260810 06:22:43.189327 22582 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6a725ccb270d476da22dceeec315bfe2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 06:22:43.223999 22621 catalog_manager.cc:1259] Loaded cluster ID: 8953f45fec83456b878495a41d75c199
I20260810 06:22:43.224073 22621 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5: loading cluster ID for follower catalog manager: success
I20260810 06:22:43.225713 22621 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5: acquiring CA information for follower catalog manager: success
I20260810 06:22:43.226888 22621 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f84c14cede214dfe9234bcc31dd0dda5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 06:22:43.519559 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 22152
I20260810 06:22:43.531617 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 22302
I20260810 06:22:43.542894 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 22428
2026-08-10T06:22:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3259 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:22:43 dist-test-slave-q9h9 krb5kdc[22653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:22:43 dist-test-slave-q9h9 krb5kdc[22653](info): set up 2 sockets
Aug 10 06:22:43 dist-test-slave-q9h9 krb5kdc[22653](info): commencing operation
krb5kdc: starting...
W20260810 06:22:44.736758 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:22:44 dist-test-slave-q9h9 krb5kdc[22653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:22:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:22:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:22:44.846010 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36183
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:36183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:22:44.949566 22673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:44.949780 22673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:44.949833 22673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:44.953653 22673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:22:44.953704 22673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:44.953730 22673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:44.953752 22673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:22:44.953778 22673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:22:44.958660 22673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:36183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:44.959489 22673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:44.960350 22673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:22:44.967878 22685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:44.967923 22682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:44.967943 22681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:44.968048 22673 server_base.cc:1034] running on GCE node
I20260810 06:22:44.968691 22673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:44.969607 22673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:44.970767 22673 hybrid_clock.cc:648] HybridClock initialized: now 1786342964970744 us; error 33 us; skew 500 ppm
Aug 10 06:22:44 dist-test-slave-q9h9 krb5kdc[22653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342964, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:44.973153 22673 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:22:44.973903 22673 webserver.cc:460] Webserver started at http://127.31.204.254:35319/ using document root <none> and password file <none>
I20260810 06:22:44.974392 22673 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:44.974457 22673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:44.974645 22673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:22:44.976432 22673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "d8d9d6609f32443f99318264641f80a2"
format_stamp: "Formatted at 2026-08-10 06:22:44 on dist-test-slave-q9h9"
server_key: "e30d694c86a93e74044f92c818643652"
server_key_iv: "41a0cc8155ff8f52d2e3f92a017403b2"
server_key_version: "encryptionkey@0"
I20260810 06:22:44.976922 22673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "d8d9d6609f32443f99318264641f80a2"
format_stamp: "Formatted at 2026-08-10 06:22:44 on dist-test-slave-q9h9"
server_key: "e30d694c86a93e74044f92c818643652"
server_key_iv: "41a0cc8155ff8f52d2e3f92a017403b2"
server_key_version: "encryptionkey@0"
I20260810 06:22:44.980289 22673 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 06:22:44.982370 22695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:44.983397 22673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:22:44.983511 22673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "d8d9d6609f32443f99318264641f80a2"
format_stamp: "Formatted at 2026-08-10 06:22:44 on dist-test-slave-q9h9"
server_key: "e30d694c86a93e74044f92c818643652"
server_key_iv: "41a0cc8155ff8f52d2e3f92a017403b2"
server_key_version: "encryptionkey@0"
I20260810 06:22:44.983604 22673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:44.997602 22673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:44.999904 22673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:45.000057 22673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:45.006330 22673 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:36183
I20260810 06:22:45.006350 22782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:36183 every 8 connection(s)
I20260810 06:22:45.007120 22673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:22:45.009536 22785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:22:45.010453 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 22673
I20260810 06:22:45.010557 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:22:45.010814 32563 external_mini_cluster.cc:1428] Setting key c9274366ac83145e2e65b8e2324e1c78
I20260810 06:22:45.015290 22785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2: Bootstrap starting.
I20260810 06:22:45.017782 22785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2: Neither blocks nor log segments found. Creating new log.
I20260810 06:22:45.018709 22785 log.cc:824] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2: Log is configured to *not* fsync() on all Append() calls
Aug 10 06:22:45 dist-test-slave-q9h9 krb5kdc[22653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:22:45.020514 22785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2: No bootstrap required, opened a new log
I20260810 06:22:45.023700 22785 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d9d6609f32443f99318264641f80a2" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36183 } }
I20260810 06:22:45.023913 22785 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:22:45.023960 22785 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d9d6609f32443f99318264641f80a2, State: Initialized, Role: FOLLOWER
I20260810 06:22:45.024230 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:45.011964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39294 (local address 127.31.204.254:36183)
0810 06:22:45.012345 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:45.012360 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:45.012394 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:22:45.013000 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:45.013014 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:45.013329 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:45.013585 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:45.014513 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:45.015502 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:45.016635 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:45.017097 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:45.017241 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:45.019976 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:45.019980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:45.019992 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:45.020017 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:45.021565 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:45.021964 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:45.021970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:45.021973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:45.022117 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:45.022381 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:45.022385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:45.022387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:45.022738 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:45.022928 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:45.023218 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 06:22:45.023469 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":122,"threads_started":1}
I20260810 06:22:45.024380 22785 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d9d6609f32443f99318264641f80a2" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36183 } }
I20260810 06:22:45.024528 22785 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:22:45.024591 22785 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:22:45.024683 22785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:22:45.025455 22785 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d9d6609f32443f99318264641f80a2" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36183 } }
I20260810 06:22:45.025766 22785 leader_election.cc:302] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8d9d6609f32443f99318264641f80a2; no voters:
I20260810 06:22:45.026103 22785 leader_election.cc:288] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:22:45.026177 22790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:22:45.026371 22790 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2 [term 1 LEADER]: Becoming Leader. State: Replica: d8d9d6609f32443f99318264641f80a2, State: Running, Role: LEADER
I20260810 06:22:45.026731 22790 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d9d6609f32443f99318264641f80a2" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36183 } }
I20260810 06:22:45.026921 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:22:45.027247 22785 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:22:45.027745 22799 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8d9d6609f32443f99318264641f80a2. Latest consensus state: current_term: 1 leader_uuid: "d8d9d6609f32443f99318264641f80a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d9d6609f32443f99318264641f80a2" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36183 } } }
I20260810 06:22:45.027853 22799 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2 [sys.catalog]: This master's current role is: LEADER
I20260810 06:22:45.028476 22791 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8d9d6609f32443f99318264641f80a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d9d6609f32443f99318264641f80a2" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36183 } } }
I20260810 06:22:45.028584 22791 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2 [sys.catalog]: This master's current role is: LEADER
I20260810 06:22:45.029655 22806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:22:45.032063 22806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:22:45.036671 22806 catalog_manager.cc:1347] Generated new cluster ID: 94a9bfd7d1f8432197e4ceb5fcf6ecab
I20260810 06:22:45.036732 22806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:22:45.063905 22806 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:22:45.064795 22806 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:22:45.070884 22806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8d9d6609f32443f99318264641f80a2: Generated new TSK 0
I20260810 06:22:45.071472 22806 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:22:45.082727 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 22673
2026-08-10T06:22:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1542 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:22:45 dist-test-slave-q9h9 krb5kdc[22838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:22:45 dist-test-slave-q9h9 krb5kdc[22838](info): set up 2 sockets
Aug 10 06:22:45 dist-test-slave-q9h9 krb5kdc[22838](info): commencing operation
krb5kdc: starting...
W20260810 06:22:46.230635 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:22:46 dist-test-slave-q9h9 krb5kdc[22838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:22:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:22:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:22:46.342713 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:46155
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:46155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:22:46.448567 22857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:46.448807 22857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:46.448843 22857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:46.452698 22857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:22:46.452754 22857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:46.452780 22857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:46.452800 22857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:22:46.452821 22857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:22:46.457799 22857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:46155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:46155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:46.458740 22857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:46.459667 22857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:22:46.467571 22872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:46.467525 22866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:46.467568 22857 server_base.cc:1034] running on GCE node
W20260810 06:22:46.467525 22868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:46.468302 22857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:46.469269 22857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:46.470419 22857 hybrid_clock.cc:648] HybridClock initialized: now 1786342966470405 us; error 29 us; skew 500 ppm
Aug 10 06:22:46 dist-test-slave-q9h9 krb5kdc[22838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342966, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:46.472932 22857 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:22:46.473753 22857 webserver.cc:460] Webserver started at http://127.31.204.254:41155/ using document root <none> and password file <none>
I20260810 06:22:46.474304 22857 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:46.474370 22857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:46.474561 22857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:22:46.476512 22857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "e0071217f0dd496e8982e7f70b22868c"
format_stamp: "Formatted at 2026-08-10 06:22:46 on dist-test-slave-q9h9"
server_key: "7880668273f7bdf91310b09fa8715431"
server_key_iv: "501d280993ecb494b5310ec25761e9eb"
server_key_version: "encryptionkey@0"
I20260810 06:22:46.477059 22857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "e0071217f0dd496e8982e7f70b22868c"
format_stamp: "Formatted at 2026-08-10 06:22:46 on dist-test-slave-q9h9"
server_key: "7880668273f7bdf91310b09fa8715431"
server_key_iv: "501d280993ecb494b5310ec25761e9eb"
server_key_version: "encryptionkey@0"
I20260810 06:22:46.480522 22857 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:22:46.482753 22886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:46.483865 22857 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:22:46.483982 22857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "e0071217f0dd496e8982e7f70b22868c"
format_stamp: "Formatted at 2026-08-10 06:22:46 on dist-test-slave-q9h9"
server_key: "7880668273f7bdf91310b09fa8715431"
server_key_iv: "501d280993ecb494b5310ec25761e9eb"
server_key_version: "encryptionkey@0"
I20260810 06:22:46.484068 22857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:46.502498 22857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:46.504951 22857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:46.505118 22857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:46.511765 22857 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:46155
I20260810 06:22:46.511767 22978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:46155 every 8 connection(s)
I20260810 06:22:46.512641 22857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:22:46.515226 22979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:22:46.517491 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 22857
I20260810 06:22:46.517618 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:22:46.517930 32563 external_mini_cluster.cc:1428] Setting key 52aa4ca859dd97d3393a9ab5825b7e1b
I20260810 06:22:46.521072 22979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c: Bootstrap starting.
I20260810 06:22:46.523533 22979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c: Neither blocks nor log segments found. Creating new log.
I20260810 06:22:46.524292 22979 log.cc:824] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c: Log is configured to *not* fsync() on all Append() calls
I20260810 06:22:46.526382 22979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c: No bootstrap required, opened a new log
Aug 10 06:22:46 dist-test-slave-q9h9 krb5kdc[22838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:22:46.530088 22979 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0071217f0dd496e8982e7f70b22868c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46155 } }
I20260810 06:22:46.530354 22979 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:22:46.530452 22979 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0071217f0dd496e8982e7f70b22868c, State: Initialized, Role: FOLLOWER
I20260810 06:22:46.530946 22979 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0071217f0dd496e8982e7f70b22868c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46155 } }
I20260810 06:22:46.531117 22979 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:22:46.531194 22979 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:22:46.531291 22979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:22:46.532328 22979 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0071217f0dd496e8982e7f70b22868c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46155 } }
I20260810 06:22:46.532666 22979 leader_election.cc:302] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0071217f0dd496e8982e7f70b22868c; no voters:
I20260810 06:22:46.532770 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:46.519234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49778 (local address 127.31.204.254:46155)
0810 06:22:46.519617 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:46.519632 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:46.519666 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:22:46.520292 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:46.520307 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:46.520609 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:46.520868 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:46.521749 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:46.522733 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:46.523847 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:46.524301 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:46.524438 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:46.527022 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:46.527028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:46.527043 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:46.527072 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:46.529712 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:46.530150 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:46.530155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:46.530161 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:46.530306 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:46.530584 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:46.530588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:46.530589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:46.531055 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:46.531251 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:46.531555 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 06:22:46.531834 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":103,"threads_started":1}
I20260810 06:22:46.532985 22979 leader_election.cc:288] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:22:46.533087 22993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:22:46.533313 22993 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 1 LEADER]: Becoming Leader. State: Replica: e0071217f0dd496e8982e7f70b22868c, State: Running, Role: LEADER
I20260810 06:22:46.533653 22993 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0071217f0dd496e8982e7f70b22868c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46155 } }
I20260810 06:22:46.534206 22979 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:22:46.534868 22993 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0071217f0dd496e8982e7f70b22868c. Latest consensus state: current_term: 1 leader_uuid: "e0071217f0dd496e8982e7f70b22868c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0071217f0dd496e8982e7f70b22868c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46155 } } }
I20260810 06:22:46.535044 22993 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [sys.catalog]: This master's current role is: LEADER
I20260810 06:22:46.535305 22995 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0071217f0dd496e8982e7f70b22868c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0071217f0dd496e8982e7f70b22868c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46155 } } }
I20260810 06:22:46.535405 22995 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [sys.catalog]: This master's current role is: LEADER
I20260810 06:22:46.535486 23008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:22:46.538537 23008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:22:46.543751 23008 catalog_manager.cc:1347] Generated new cluster ID: 313efae9ae0f46ccbe9515d652c6af58
I20260810 06:22:46.543802 23008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:22:46.551050 23008 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:22:46.551860 23008 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:22:46.557322 23008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c: Generated new TSK 0
I20260810 06:22:46.557976 23008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
I20260810 06:22:46.590122 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:46155
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:22:46.699647 23026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:46.699873 23026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:46.699913 23026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:46.703709 23026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:46.703752 23026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:46.703805 23026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:22:46.708705 23026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:46.709578 23026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:46.710515 23026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:22:46.719506 23026 server_base.cc:1034] running on GCE node
W20260810 06:22:46.719573 23038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:46.719506 23040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:46.719488 23036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:46.720086 23026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:46.720649 23026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:46.721792 23026 hybrid_clock.cc:648] HybridClock initialized: now 1786342966721768 us; error 37 us; skew 500 ppm
Aug 10 06:22:46 dist-test-slave-q9h9 krb5kdc[22838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342966, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:46.724426 23026 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:22:46.725214 23026 webserver.cc:460] Webserver started at http://127.31.204.193:41079/ using document root <none> and password file <none>
I20260810 06:22:46.725718 23026 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:46.725780 23026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:46.725966 23026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:22:46.727878 23026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "befeb79e90b2436ea19b3a412dfe0c0a"
format_stamp: "Formatted at 2026-08-10 06:22:46 on dist-test-slave-q9h9"
server_key: "1dd3b43790a84bfbcadf562f8869396c"
server_key_iv: "166f7ee6c33de3a785ed77424662179b"
server_key_version: "encryptionkey@0"
I20260810 06:22:46.728390 23026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "befeb79e90b2436ea19b3a412dfe0c0a"
format_stamp: "Formatted at 2026-08-10 06:22:46 on dist-test-slave-q9h9"
server_key: "1dd3b43790a84bfbcadf562f8869396c"
server_key_iv: "166f7ee6c33de3a785ed77424662179b"
server_key_version: "encryptionkey@0"
I20260810 06:22:46.731797 23026 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:22:46.733923 23052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:46.735098 23026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:22:46.735204 23026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "befeb79e90b2436ea19b3a412dfe0c0a"
format_stamp: "Formatted at 2026-08-10 06:22:46 on dist-test-slave-q9h9"
server_key: "1dd3b43790a84bfbcadf562f8869396c"
server_key_iv: "166f7ee6c33de3a785ed77424662179b"
server_key_version: "encryptionkey@0"
I20260810 06:22:46.735301 23026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:46.749372 23026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:46.751832 23026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:46.751992 23026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:46.752535 23026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:22:46.753283 23026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:22:46.753327 23026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:46.753376 23026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:22:46.753404 23026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:46.761768 23026 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:43547
I20260810 06:22:46.761796 23239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:43547 every 8 connection(s)
I20260810 06:22:46.762609 23026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:22:46.765931 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 23026
I20260810 06:22:46.766070 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:22:46.766332 32563 external_mini_cluster.cc:1428] Setting key 37f99e1dba8261d1e0f57c05a2431346
Aug 10 06:22:46 dist-test-slave-q9h9 krb5kdc[22838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342966, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:22:46.775732 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:46.764277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:44017 (local address 127.31.204.254:46155)
0810 06:22:46.764400 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:46.764404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:46.765170 (+ 766us) server_negotiation.cc:407] Connection header received
0810 06:22:46.766515 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:46.766518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:46.766574 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:46.766646 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:46.767870 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:46.768393 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:46.769487 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:46.769835 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:46.769930 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:46.772254 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:46.772257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:46.772260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:46.772290 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:46.773787 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:46.774285 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:46.774288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:46.774290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:46.774381 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:46.774669 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:46.774673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:46.774674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:46.774916 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:46.775014 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:46.775445 (+ 431us) server_negotiation.cc:299] Negotiation successful
0810 06:22:46.775570 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.31.204.194@KRBTEST.COM; defaulting to no policy
I20260810 06:22:46.776453 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:46.764590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:44017)
0810 06:22:46.765034 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:46.765074 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:46.766320 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:46.766801 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:46.766807 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:46.767226 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:46.767739 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:46.767753 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:46.768508 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:46.768511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:46.769374 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:46.769383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:46.769995 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:46.769998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:46.770185 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:46.770715 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:46.770730 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:46.772080 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:46.773897 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:46.773901 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:46.773912 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:46.774198 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:46.774462 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:46.774464 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:46.774465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:46.774584 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:46.775000 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:46.775005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:46.775214 (+ 209us) client_negotiation.cc:769] Sending connection context
0810 06:22:46.775351 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 06:22:46.775578 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":111,"threads_started":1}
I20260810 06:22:46.777556 23240 heartbeater.cc:347] Connected to a master server at 127.31.204.254:46155
I20260810 06:22:46.777757 23240 heartbeater.cc:464] Registering TS with master...
I20260810 06:22:46.778258 23240 heartbeater.cc:511] Master 127.31.204.254:46155 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.194@KRBTEST.COM" created.
I20260810 06:22:46.779938 22913 ts_manager.cc:194] Registered new tserver with Master: befeb79e90b2436ea19b3a412dfe0c0a (127.31.204.193:43547)
I20260810 06:22:46.781205 22913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:44017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
I20260810 06:22:46.792850 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:46155
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:22:46.899407 23251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:46.899690 23251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:46.899768 23251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:46.903662 23251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:46.903712 23251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:46.903788 23251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:22:46.908776 23251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:46.909744 23251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:46.910768 23251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:22:46.920079 23264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:46.920130 23251 server_base.cc:1034] running on GCE node
W20260810 06:22:46.920087 23263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:46.920079 23266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:46.920734 23251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:46.921348 23251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:46.922503 23251 hybrid_clock.cc:648] HybridClock initialized: now 1786342966922483 us; error 29 us; skew 500 ppm
Aug 10 06:22:46 dist-test-slave-q9h9 krb5kdc[22838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342966, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:46.925115 23251 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:22:46.925983 23251 webserver.cc:460] Webserver started at http://127.31.204.194:36507/ using document root <none> and password file <none>
I20260810 06:22:46.926542 23251 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:46.926610 23251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:46.926811 23251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:22:46.928835 23251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "807c1bfaa46c4ed098402fdf225ed4c6"
format_stamp: "Formatted at 2026-08-10 06:22:46 on dist-test-slave-q9h9"
server_key: "ad67e6329f60a25f758907e4ebd25c9e"
server_key_iv: "391b85b05fa64f2ee2615e3d9b3b18d7"
server_key_version: "encryptionkey@0"
I20260810 06:22:46.929376 23251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "807c1bfaa46c4ed098402fdf225ed4c6"
format_stamp: "Formatted at 2026-08-10 06:22:46 on dist-test-slave-q9h9"
server_key: "ad67e6329f60a25f758907e4ebd25c9e"
server_key_iv: "391b85b05fa64f2ee2615e3d9b3b18d7"
server_key_version: "encryptionkey@0"
I20260810 06:22:46.932898 23251 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260810 06:22:46.935325 23278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:46.936488 23251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:22:46.936592 23251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "807c1bfaa46c4ed098402fdf225ed4c6"
format_stamp: "Formatted at 2026-08-10 06:22:46 on dist-test-slave-q9h9"
server_key: "ad67e6329f60a25f758907e4ebd25c9e"
server_key_iv: "391b85b05fa64f2ee2615e3d9b3b18d7"
server_key_version: "encryptionkey@0"
I20260810 06:22:46.936684 23251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:46.954448 23251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:46.957465 23251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:46.957643 23251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:46.958273 23251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:22:46.959081 23251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:22:46.959129 23251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:46.959182 23251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:22:46.959218 23251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:46.967523 23251 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:37759
I20260810 06:22:46.967551 23476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:37759 every 8 connection(s)
I20260810 06:22:46.968403 23251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
Aug 10 06:22:46 dist-test-slave-q9h9 krb5kdc[22838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342966, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:22:46.978525 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 23251
I20260810 06:22:46.978659 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:22:46.978995 32563 external_mini_cluster.cc:1428] Setting key 874dcc18b54a88755fa32dcec1f876b4
I20260810 06:22:46.981338 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:46.970150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:47723 (local address 127.31.204.254:46155)
0810 06:22:46.970285 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:46.970289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:46.971091 (+ 802us) server_negotiation.cc:407] Connection header received
0810 06:22:46.972404 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:46.972407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:46.972464 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:46.972536 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:46.973653 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:46.974211 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:46.975334 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:46.975556 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:46.975645 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:46.977889 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:46.977892 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:46.977893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:46.977918 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:46.979381 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:46.979856 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:46.979860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:46.979862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:46.979943 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:46.980247 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:46.980250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:46.980252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:46.980468 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:46.980569 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:46.981062 (+ 493us) server_negotiation.cc:299] Negotiation successful
0810 06:22:46.981177 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:22:46.982101 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:46.970496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:47723)
0810 06:22:46.970952 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:46.970992 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:46.972197 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:46.972712 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:46.972718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:46.973057 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:46.973508 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:46.973521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:46.974325 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:46.974329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:46.975211 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:46.975221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:46.975696 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:46.975700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:46.975866 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:46.976351 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:46.976366 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:46.977723 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:46.979496 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:46.979502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:46.979516 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:46.979755 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:46.980023 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:46.980026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:46.980028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:46.980159 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:46.980582 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:46.980587 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:46.980823 (+ 236us) client_negotiation.cc:769] Sending connection context
0810 06:22:46.980980 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 06:22:46.981237 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":118,"threads_started":1}
I20260810 06:22:46.983268 23477 heartbeater.cc:347] Connected to a master server at 127.31.204.254:46155
I20260810 06:22:46.983482 23477 heartbeater.cc:464] Registering TS with master...
I20260810 06:22:46.983951 23477 heartbeater.cc:511] Master 127.31.204.254:46155 requested a full tablet report, sending...
I20260810 06:22:46.984992 22913 ts_manager.cc:194] Registered new tserver with Master: 807c1bfaa46c4ed098402fdf225ed4c6 (127.31.204.194:37759)
I20260810 06:22:46.985560 22913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:47723
WARNING: no policy specified for kudu/127.31.204.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
I20260810 06:22:47.004664 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:46155
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:22:47.110839 23484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:47.111048 23484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:47.111086 23484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:47.114945 23484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:47.114993 23484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:47.115072 23484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:22:47.120128 23484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:47.121022 23484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:47.121937 23484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:22:47.130687 23495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:47.130816 23484 server_base.cc:1034] running on GCE node
W20260810 06:22:47.130740 23493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:47.130740 23490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:47.131366 23484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:47.131947 23484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:47.133095 23484 hybrid_clock.cc:648] HybridClock initialized: now 1786342967133078 us; error 30 us; skew 500 ppm
Aug 10 06:22:47 dist-test-slave-q9h9 krb5kdc[22838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342967, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:47.135732 23484 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:22:47.136539 23484 webserver.cc:460] Webserver started at http://127.31.204.195:32953/ using document root <none> and password file <none>
I20260810 06:22:47.137048 23484 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:47.137110 23484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:47.137286 23484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:22:47.139247 23484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "bac5e0a95e9442bfa498f54d0b98c598"
format_stamp: "Formatted at 2026-08-10 06:22:47 on dist-test-slave-q9h9"
server_key: "dff70594f592cc55ef9143b416fb6b18"
server_key_iv: "438bb2f9530a16cccacfb492a80a1319"
server_key_version: "encryptionkey@0"
I20260810 06:22:47.139806 23484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "bac5e0a95e9442bfa498f54d0b98c598"
format_stamp: "Formatted at 2026-08-10 06:22:47 on dist-test-slave-q9h9"
server_key: "dff70594f592cc55ef9143b416fb6b18"
server_key_iv: "438bb2f9530a16cccacfb492a80a1319"
server_key_version: "encryptionkey@0"
I20260810 06:22:47.143246 23484 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:22:47.145358 23505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:47.146488 23484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:22:47.146591 23484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "bac5e0a95e9442bfa498f54d0b98c598"
format_stamp: "Formatted at 2026-08-10 06:22:47 on dist-test-slave-q9h9"
server_key: "dff70594f592cc55ef9143b416fb6b18"
server_key_iv: "438bb2f9530a16cccacfb492a80a1319"
server_key_version: "encryptionkey@0"
I20260810 06:22:47.146679 23484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:47.164113 23484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:47.167013 23484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:47.167182 23484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:47.167723 23484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:22:47.168480 23484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:22:47.168526 23484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:47.168576 23484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:22:47.168606 23484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:47.176425 23484 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:38687
I20260810 06:22:47.176468 23696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:38687 every 8 connection(s)
I20260810 06:22:47.177245 23484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:22:47.180083 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 23484
I20260810 06:22:47.180194 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:22:47.180481 32563 external_mini_cluster.cc:1428] Setting key f5dd2fbedfb8e67fc5bb699e3cd14132
Aug 10 06:22:47 dist-test-slave-q9h9 krb5kdc[22838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342967, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:22:47.189803 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.178942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:46803 (local address 127.31.204.254:46155)
0810 06:22:47.179075 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:47.179080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:47.179877 (+ 797us) server_negotiation.cc:407] Connection header received
0810 06:22:47.181456 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:47.181458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:47.181504 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:47.181569 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:47.182827 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.183351 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.184377 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.184572 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.184636 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.186799 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:47.186802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:47.186804 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:47.186829 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:47.187970 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:47.188450 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:47.188454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:47.188455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:47.188530 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:47.188831 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:47.188834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:47.188836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:47.189002 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:47.189085 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:47.189528 (+ 443us) server_negotiation.cc:299] Negotiation successful
0810 06:22:47.189650 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:22:47.190539 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.179306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:46803)
0810 06:22:47.179731 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:47.179773 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:47.181253 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:47.181754 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:47.181762 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:47.182168 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:47.182707 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.182720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.183443 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.183445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.184292 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.184301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.184670 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.184672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.184804 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.185247 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:47.185261 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:47.186652 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:47.188074 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:47.188080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:47.188093 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:47.188362 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:47.188610 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:47.188613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:47.188615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:47.188740 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:47.189097 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:47.189103 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:47.189316 (+ 213us) client_negotiation.cc:769] Sending connection context
0810 06:22:47.189471 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 06:22:47.189753 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":107,"threads_started":1}
I20260810 06:22:47.191462 23697 heartbeater.cc:347] Connected to a master server at 127.31.204.254:46155
I20260810 06:22:47.191669 23697 heartbeater.cc:464] Registering TS with master...
I20260810 06:22:47.192098 23697 heartbeater.cc:511] Master 127.31.204.254:46155 requested a full tablet report, sending...
I20260810 06:22:47.193009 22913 ts_manager.cc:194] Registered new tserver with Master: bac5e0a95e9442bfa498f54d0b98c598 (127.31.204.195:38687)
I20260810 06:22:47.193567 22913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:46803
I20260810 06:22:47.193976 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:22:47.201639 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.195827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49782 (local address 127.31.204.254:46155)
0810 06:22:47.195922 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:47.195924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:47.196095 (+ 171us) server_negotiation.cc:407] Connection header received
0810 06:22:47.196189 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:47.196191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:47.196221 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:47.196289 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:47.196851 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.197309 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.198034 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.198251 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.198327 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.199405 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:47.199408 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:47.199409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:47.199423 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:47.200458 (+ 1035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:47.200806 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:47.200808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:47.200809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:47.200866 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:47.201036 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:47.201038 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:47.201039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:47.201211 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:47.201316 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:47.201462 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 06:22:47.201539 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 06:22:47.203924 22913 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:22:47.206703 22913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:22:47.219939 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.215102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38358 (local address 127.31.204.193:43547)
0810 06:22:47.215437 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:47.215447 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:47.215659 (+ 212us) server_negotiation.cc:407] Connection header received
0810 06:22:47.215902 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:47.215908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:47.216077 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:47.216240 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:22:47.216401 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.217444 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.218737 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.219323 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.219430 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.219563 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:47.219709 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 06:22:47.219800 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":111,"threads_started":1}
I20260810 06:22:47.219939 23731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.214651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:38687 (local address 127.0.0.1:37946)
0810 06:22:47.215118 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:47.215130 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:47.215207 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:47.215853 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:47.215858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:47.215880 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:22:47.216041 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.216048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.217126 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.217129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.218410 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.218421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.219477 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.219481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.219577 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.219602 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 06:22:47.219646 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 06:22:47.219791 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260810 06:22:47.220263 23733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.215003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.31.204.195:38687)
0810 06:22:47.215404 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:47.215415 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:47.215436 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:22:47.215502 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:47.215507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:47.215697 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:47.216184 (+ 487us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:22:47.216320 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.217011 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.218540 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.219338 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.219885 (+ 547us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.220005 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:47.220053 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 06:22:47.220134 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":88,"threads_started":1}
I20260810 06:22:47.219947 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.214273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:37759 (local address 127.0.0.1:52718)
0810 06:22:47.214730 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:47.214789 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:47.214916 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:47.215909 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:47.215912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:47.215925 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:22:47.216042 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.216048 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.217421 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.217424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.218576 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.218584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.219615 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.219618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.219695 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.219715 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 06:22:47.219755 (+ 40us) client_negotiation.cc:240] Negotiation successful
0810 06:22:47.219811 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":64,"thread_start_us":51,"threads_started":1}
I20260810 06:22:47.220388 23726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.214447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52718 (local address 127.31.204.194:37759)
0810 06:22:47.215434 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:47.215449 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:47.215472 (+ 23us) server_negotiation.cc:407] Connection header received
0810 06:22:47.215545 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:47.215550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:47.215754 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:47.215944 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:22:47.216154 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.217291 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.218695 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.219500 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.220027 (+ 527us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.220132 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:47.220175 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 06:22:47.220247 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":126,"threads_started":1}
I20260810 06:22:47.220522 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.214933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:43547 (local address 127.0.0.1:38358)
0810 06:22:47.215571 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:47.215583 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:47.215720 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:47.216199 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:47.216201 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:47.216217 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:22:47.216288 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.216293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.217560 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.217563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.218625 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.218635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.219429 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.219433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.219536 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.219571 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 06:22:47.219671 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 06:22:47.219787 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":41,"thread_start_us":89,"threads_started":1}
I20260810 06:22:47.222796 23586 tablet_service.cc:1467] Processing CreateTablet for tablet bb27a68d99da486fbc522654d963573d (DEFAULT_TABLE table=test-table [id=6906bef31f124f24b78fa45ae35090a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:22:47.223228 23135 tablet_service.cc:1467] Processing CreateTablet for tablet bb27a68d99da486fbc522654d963573d (DEFAULT_TABLE table=test-table [id=6906bef31f124f24b78fa45ae35090a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:22:47.223230 23364 tablet_service.cc:1467] Processing CreateTablet for tablet bb27a68d99da486fbc522654d963573d (DEFAULT_TABLE table=test-table [id=6906bef31f124f24b78fa45ae35090a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:22:47.223681 23586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb27a68d99da486fbc522654d963573d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:22:47.224237 23135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb27a68d99da486fbc522654d963573d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:22:47.224237 23364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb27a68d99da486fbc522654d963573d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:22:47.230360 23736 tablet_bootstrap.cc:492] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598: Bootstrap starting.
I20260810 06:22:47.232110 23737 tablet_bootstrap.cc:492] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6: Bootstrap starting.
I20260810 06:22:47.232729 23736 tablet_bootstrap.cc:654] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598: Neither blocks nor log segments found. Creating new log.
I20260810 06:22:47.233882 23736 log.cc:824] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598: Log is configured to *not* fsync() on all Append() calls
I20260810 06:22:47.234551 23737 tablet_bootstrap.cc:654] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6: Neither blocks nor log segments found. Creating new log.
I20260810 06:22:47.234632 23738 tablet_bootstrap.cc:492] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a: Bootstrap starting.
I20260810 06:22:47.235380 23737 log.cc:824] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6: Log is configured to *not* fsync() on all Append() calls
I20260810 06:22:47.236347 23738 tablet_bootstrap.cc:654] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a: Neither blocks nor log segments found. Creating new log.
I20260810 06:22:47.237149 23737 tablet_bootstrap.cc:492] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6: No bootstrap required, opened a new log
I20260810 06:22:47.237306 23738 log.cc:824] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a: Log is configured to *not* fsync() on all Append() calls
I20260810 06:22:47.237334 23737 ts_tablet_manager.cc:1397] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260810 06:22:47.238775 23736 tablet_bootstrap.cc:492] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598: No bootstrap required, opened a new log
I20260810 06:22:47.238974 23736 ts_tablet_manager.cc:1397] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260810 06:22:47.239466 23738 tablet_bootstrap.cc:492] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a: No bootstrap required, opened a new log
I20260810 06:22:47.239768 23738 ts_tablet_manager.cc:1397] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260810 06:22:47.241466 23737 raft_consensus.cc:348] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } } peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } }
I20260810 06:22:47.241747 23737 raft_consensus.cc:374] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:22:47.241815 23737 raft_consensus.cc:729] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 807c1bfaa46c4ed098402fdf225ed4c6, State: Initialized, Role: FOLLOWER
I20260810 06:22:47.241942 23736 raft_consensus.cc:348] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } } peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } }
I20260810 06:22:47.242204 23736 raft_consensus.cc:374] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:22:47.242259 23736 raft_consensus.cc:729] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac5e0a95e9442bfa498f54d0b98c598, State: Initialized, Role: FOLLOWER
I20260810 06:22:47.242290 23737 consensus_queue.cc:260] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } } peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } }
I20260810 06:22:47.242683 23736 consensus_queue.cc:260] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } } peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } }
I20260810 06:22:47.243046 23477 heartbeater.cc:503] Master 127.31.204.254:46155 was elected leader, sending a full tablet report...
I20260810 06:22:47.243356 23737 ts_tablet_manager.cc:1428] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 06:22:47.243605 23736 ts_tablet_manager.cc:1428] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 06:22:47.243571 23738 raft_consensus.cc:348] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } } peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } }
I20260810 06:22:47.243759 23738 raft_consensus.cc:374] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:22:47.243796 23738 raft_consensus.cc:729] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: befeb79e90b2436ea19b3a412dfe0c0a, State: Initialized, Role: FOLLOWER
I20260810 06:22:47.243945 23697 heartbeater.cc:503] Master 127.31.204.254:46155 was elected leader, sending a full tablet report...
I20260810 06:22:47.244228 23738 consensus_queue.cc:260] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } } peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } }
I20260810 06:22:47.245137 23738 ts_tablet_manager.cc:1428] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 06:22:47.245473 23240 heartbeater.cc:503] Master 127.31.204.254:46155 was elected leader, sending a full tablet report...
W20260810 06:22:47.263612 23246 tablet.cc:2369] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:22:47.276440 23748 raft_consensus.cc:482] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:22:47.276603 23748 raft_consensus.cc:504] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } } peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } }
I20260810 06:22:47.277669 23748 leader_election.cc:288] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 807c1bfaa46c4ed098402fdf225ed4c6 (127.31.204.194:37759), befeb79e90b2436ea19b3a412dfe0c0a (127.31.204.193:43547)
I20260810 06:22:47.281026 23726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.277978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:45401 (local address 127.31.204.194:37759)
0810 06:22:47.278108 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:47.278113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:47.278128 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:22:47.278207 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:47.278210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:47.278258 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:47.278343 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:22:47.278597 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.279093 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.280114 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.280520 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.280625 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.280681 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:47.280848 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 06:22:47.280920 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 06:22:47.281026 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.277875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:37759 (local address 127.31.204.195:45401)
0810 06:22:47.277998 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:47.278034 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:47.278119 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:47.278362 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:47.278365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:47.278386 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:22:47.278495 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.278501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.279199 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.279203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.279997 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.280006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.280627 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.280631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.280735 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.280753 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 06:22:47.280824 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 06:22:47.280896 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":23}
I20260810 06:22:47.281481 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.278258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:39363 (local address 127.31.204.193:43547)
0810 06:22:47.278392 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:47.278397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:47.278610 (+ 213us) server_negotiation.cc:407] Connection header received
0810 06:22:47.278711 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:47.278714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:47.278757 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:47.278818 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:22:47.279071 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.279553 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.280579 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.280954 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.281058 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.281108 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:47.281292 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 06:22:47.281370 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:22:47.281477 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.278156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:43547 (local address 127.31.204.195:39363)
0810 06:22:47.278526 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:47.278537 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:47.278628 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:47.278834 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:47.278837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:47.278852 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:22:47.278970 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.278975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.279652 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.279656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.280458 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.280467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.281074 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.281079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.281169 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.281192 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 06:22:47.281265 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 06:22:47.281329 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":22,"thread_start_us":104,"threads_started":1}
I20260810 06:22:47.281678 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb27a68d99da486fbc522654d963573d" candidate_uuid: "bac5e0a95e9442bfa498f54d0b98c598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" is_pre_election: true
I20260810 06:22:47.282050 23394 raft_consensus.cc:2454] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bac5e0a95e9442bfa498f54d0b98c598 in term 0.
I20260810 06:22:47.282123 23165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb27a68d99da486fbc522654d963573d" candidate_uuid: "bac5e0a95e9442bfa498f54d0b98c598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" is_pre_election: true
I20260810 06:22:47.282390 23165 raft_consensus.cc:2454] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bac5e0a95e9442bfa498f54d0b98c598 in term 0.
I20260810 06:22:47.282591 23515 leader_election.cc:302] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 807c1bfaa46c4ed098402fdf225ed4c6, bac5e0a95e9442bfa498f54d0b98c598; no voters:
I20260810 06:22:47.282862 23748 raft_consensus.cc:2781] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:22:47.282928 23748 raft_consensus.cc:482] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:22:47.282955 23748 raft_consensus.cc:3037] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:22:47.283938 23748 raft_consensus.cc:504] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } } peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } }
I20260810 06:22:47.284245 23748 leader_election.cc:288] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [CANDIDATE]: Term 1 election: Requested vote from peers 807c1bfaa46c4ed098402fdf225ed4c6 (127.31.204.194:37759), befeb79e90b2436ea19b3a412dfe0c0a (127.31.204.193:43547)
I20260810 06:22:47.284555 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb27a68d99da486fbc522654d963573d" candidate_uuid: "bac5e0a95e9442bfa498f54d0b98c598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "807c1bfaa46c4ed098402fdf225ed4c6"
I20260810 06:22:47.284621 23165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb27a68d99da486fbc522654d963573d" candidate_uuid: "bac5e0a95e9442bfa498f54d0b98c598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "befeb79e90b2436ea19b3a412dfe0c0a"
I20260810 06:22:47.284662 23394 raft_consensus.cc:3037] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:22:47.284735 23165 raft_consensus.cc:3037] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:22:47.285656 23394 raft_consensus.cc:2454] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bac5e0a95e9442bfa498f54d0b98c598 in term 1.
I20260810 06:22:47.285722 23165 raft_consensus.cc:2454] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bac5e0a95e9442bfa498f54d0b98c598 in term 1.
I20260810 06:22:47.285904 23515 leader_election.cc:302] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 807c1bfaa46c4ed098402fdf225ed4c6, bac5e0a95e9442bfa498f54d0b98c598; no voters:
I20260810 06:22:47.286083 23748 raft_consensus.cc:2781] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:22:47.286307 23748 raft_consensus.cc:686] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [term 1 LEADER]: Becoming Leader. State: Replica: bac5e0a95e9442bfa498f54d0b98c598, State: Running, Role: LEADER
I20260810 06:22:47.286594 23748 consensus_queue.cc:237] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } } peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } }
I20260810 06:22:47.289481 22911 catalog_manager.cc:5495] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 reported cstate change: term changed from 0 to 1, leader changed from <none> to bac5e0a95e9442bfa498f54d0b98c598 (127.31.204.195). New cstate: current_term: 1 leader_uuid: "bac5e0a95e9442bfa498f54d0b98c598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } health_report { overall_health: HEALTHY } } }
I20260810 06:22:47.304682 23733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.301796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37952 (local address 127.31.204.195:38687)
0810 06:22:47.301911 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:47.301914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:47.301926 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:22:47.302073 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:47.302075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:47.302123 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:47.302202 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:22:47.302384 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.302845 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.303576 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.303765 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.303824 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.304030 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:22:47.304407 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:22:47.304488 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:47.304521 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 06:22:47.304564 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:22:47.309948 23165 raft_consensus.cc:1261] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Refusing update from remote peer bac5e0a95e9442bfa498f54d0b98c598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:22:47.309957 23394 raft_consensus.cc:1261] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6 [term 1 FOLLOWER]: Refusing update from remote peer bac5e0a95e9442bfa498f54d0b98c598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:22:47.310617 23748 consensus_queue.cc:1035] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:22:47.310809 23754 consensus_queue.cc:1035] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:22:47.316979 23759 mvcc.cc:205] Tried to move back new op lower bound from 7316860794095263744 to 7316860794008555520. Current Snapshot: MvccSnapshot[applied={T|T < 7316860794095263744}]
I20260810 06:22:47.318127 23762 mvcc.cc:205] Tried to move back new op lower bound from 7316860794095263744 to 7316860794008555520. Current Snapshot: MvccSnapshot[applied={T|T < 7316860794095263744}]
I20260810 06:22:47.320519 23763 mvcc.cc:205] Tried to move back new op lower bound from 7316860794095263744 to 7316860794008555520. Current Snapshot: MvccSnapshot[applied={T|T < 7316860794095263744 or (T in {7316860794095263744})}]
I20260810 06:22:47.325450 22911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:22:47.325582 22911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:22:47.327854 22911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c: Sending DeleteTablet for 3 replicas of tablet bb27a68d99da486fbc522654d963573d
I20260810 06:22:47.328606 23586 tablet_service.cc:1514] Processing DeleteTablet for tablet bb27a68d99da486fbc522654d963573d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:22:47 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:37946
I20260810 06:22:47.328631 23364 tablet_service.cc:1514] Processing DeleteTablet for tablet bb27a68d99da486fbc522654d963573d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:22:47 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:52718
I20260810 06:22:47.328709 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 23026
I20260810 06:22:47.338377 23772 tablet_replica.cc:330] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598: stopping tablet replica
I20260810 06:22:47.338742 23772 raft_consensus.cc:2229] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:22:47.339251 23772 raft_consensus.cc:2258] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:22:47.340939 23772 ts_tablet_manager.cc:1905] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:22:47.343741 23772 ts_tablet_manager.cc:1918] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:22:47.343848 23772 log.cc:1196] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/bb27a68d99da486fbc522654d963573d
I20260810 06:22:47.344177 23772 ts_tablet_manager.cc:1939] T bb27a68d99da486fbc522654d963573d P bac5e0a95e9442bfa498f54d0b98c598: Deleting consensus metadata
I20260810 06:22:47.345381 22892 catalog_manager.cc:4841] TS bac5e0a95e9442bfa498f54d0b98c598 (127.31.204.195:38687): tablet bb27a68d99da486fbc522654d963573d (table test-table [id=6906bef31f124f24b78fa45ae35090a6]) successfully deleted
I20260810 06:22:47.345600 23773 tablet_replica.cc:330] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6: stopping tablet replica
I20260810 06:22:47.345932 23773 raft_consensus.cc:2229] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:22:47.346278 23773 raft_consensus.cc:2258] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:22:47.347613 23773 ts_tablet_manager.cc:1905] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 06:22:47.347970 22893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.204.193:43547 (error 108)
I20260810 06:22:47.348259 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 23251
I20260810 06:22:47.354830 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.350191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:43547 (local address 127.0.0.1:38360)
0810 06:22:47.354645 (+ 4454us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:47.354725 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4355}
W20260810 06:22:47.358942 22893 catalog_manager.cc:4584] TS befeb79e90b2436ea19b3a412dfe0c0a (127.31.204.193:43547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb27a68d99da486fbc522654d963573d: Network error: Client connection negotiation failed: client connection to 127.31.204.193:43547: connect: Connection refused (error 111)
I20260810 06:22:47.373869 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 23484
I20260810 06:22:47.373915 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.373631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:37759 (local address 127.0.0.1:52732)
0810 06:22:47.373755 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:47.373822 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.194:37759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 06:22:47.382723 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 22857
I20260810 06:22:47.396436 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:46155
--webserver_interface=127.31.204.254
--webserver_port=41155
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:46155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:22:47.502779 23775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:47.502998 23775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:47.503037 23775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:47.506892 23775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:22:47.506937 23775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:47.506956 23775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:47.506985 23775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:22:47.507011 23775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:22:47.512040 23775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:46155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:46155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=41155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:47.512904 23775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:47.513827 23775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:22:47.527946 23790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:47.528035 23775 server_base.cc:1034] running on GCE node
W20260810 06:22:47.527896 23789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:47.527896 23795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:47.528735 23775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:47.529739 23775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:47.530889 23775 hybrid_clock.cc:648] HybridClock initialized: now 1786342967530872 us; error 32 us; skew 500 ppm
Aug 10 06:22:47 dist-test-slave-q9h9 krb5kdc[22838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342967, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:47.538879 23775 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:22:47.539726 23775 webserver.cc:460] Webserver started at http://127.31.204.254:41155/ using document root <none> and password file <none>
I20260810 06:22:47.540249 23775 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:47.540335 23775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:47.543994 23775 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:22:47.545951 23807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:47.547078 23775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:22:47.547190 23775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "e0071217f0dd496e8982e7f70b22868c"
format_stamp: "Formatted at 2026-08-10 06:22:46 on dist-test-slave-q9h9"
server_key: "7880668273f7bdf91310b09fa8715431"
server_key_iv: "501d280993ecb494b5310ec25761e9eb"
server_key_version: "encryptionkey@0"
I20260810 06:22:47.547750 23775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:47.563439 23775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:47.565778 23775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:47.565934 23775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:47.572329 23775 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:46155
I20260810 06:22:47.572566 23906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:46155 every 8 connection(s)
I20260810 06:22:47.573316 23775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:22:47.577435 23907 sys_catalog.cc:263] Verifying existing consensus state
I20260810 06:22:47.579061 23907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c: Bootstrap starting.
I20260810 06:22:47.582252 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 23775
I20260810 06:22:47.582638 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:43547
--local_ip_for_outbound_sockets=127.31.204.193
--tserver_master_addrs=127.31.204.254:46155
--webserver_port=41079
--webserver_interface=127.31.204.193
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:22:47.596381 23907 log.cc:824] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c: Log is configured to *not* fsync() on all Append() calls
I20260810 06:22:47.604664 23907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:22:47.604972 23907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c: Bootstrap complete.
I20260810 06:22:47.608800 23907 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0071217f0dd496e8982e7f70b22868c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46155 } }
I20260810 06:22:47.609171 23907 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0071217f0dd496e8982e7f70b22868c, State: Initialized, Role: FOLLOWER
I20260810 06:22:47.609690 23907 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0071217f0dd496e8982e7f70b22868c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46155 } }
I20260810 06:22:47.609820 23907 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:22:47.609892 23907 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:22:47.609987 23907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 1 FOLLOWER]: Advancing to term 2
I20260810 06:22:47.611780 23907 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0071217f0dd496e8982e7f70b22868c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46155 } }
I20260810 06:22:47.612082 23907 leader_election.cc:302] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0071217f0dd496e8982e7f70b22868c; no voters:
I20260810 06:22:47.612376 23907 leader_election.cc:288] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 06:22:47.612470 23911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 2 FOLLOWER]: Leader election won for term 2
I20260810 06:22:47.612673 23911 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 2 LEADER]: Becoming Leader. State: Replica: e0071217f0dd496e8982e7f70b22868c, State: Running, Role: LEADER
I20260810 06:22:47.612962 23911 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0071217f0dd496e8982e7f70b22868c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46155 } }
I20260810 06:22:47.613574 23907 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:22:47.613870 23911 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0071217f0dd496e8982e7f70b22868c. Latest consensus state: current_term: 2 leader_uuid: "e0071217f0dd496e8982e7f70b22868c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0071217f0dd496e8982e7f70b22868c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46155 } } }
I20260810 06:22:47.614087 23911 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [sys.catalog]: This master's current role is: LEADER
I20260810 06:22:47.615149 23915 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e0071217f0dd496e8982e7f70b22868c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0071217f0dd496e8982e7f70b22868c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46155 } } }
I20260810 06:22:47.615319 23915 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [sys.catalog]: This master's current role is: LEADER
I20260810 06:22:47.615394 23918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:22:47.620775 23918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:22:47.621505 23918 catalog_manager.cc:1259] Loaded cluster ID: 313efae9ae0f46ccbe9515d652c6af58
I20260810 06:22:47.621583 23918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:22:47.623064 23918 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 06:22:47.629119 23918 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 06:22:47.635959 23918 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:22:47.636644 23918 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c: Loaded TSK: 0
I20260810 06:22:47.637809 23918 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 06:22:47.706467 23909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:47.706698 23909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:47.706755 23909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:47.710660 23909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:47.710711 23909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:47.710790 23909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:22:47.715862 23909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:43547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=41079
--tserver_master_addrs=127.31.204.254:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:47.716758 23909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:47.717702 23909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:22:47.726871 23953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:47.726876 23955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:47.726943 23959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:47.727484 23909 server_base.cc:1034] running on GCE node
I20260810 06:22:47.727841 23909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:47.728418 23909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:47.729588 23909 hybrid_clock.cc:648] HybridClock initialized: now 1786342967729566 us; error 43 us; skew 500 ppm
Aug 10 06:22:47 dist-test-slave-q9h9 krb5kdc[22838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342967, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:47.732173 23909 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:22:47.732987 23909 webserver.cc:460] Webserver started at http://127.31.204.193:41079/ using document root <none> and password file <none>
I20260810 06:22:47.733487 23909 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:47.733556 23909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:47.737303 23909 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:22:47.739321 23973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:47.740463 23909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:22:47.740553 23909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "befeb79e90b2436ea19b3a412dfe0c0a"
format_stamp: "Formatted at 2026-08-10 06:22:46 on dist-test-slave-q9h9"
server_key: "1dd3b43790a84bfbcadf562f8869396c"
server_key_iv: "166f7ee6c33de3a785ed77424662179b"
server_key_version: "encryptionkey@0"
I20260810 06:22:47.741068 23909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:47.752326 23909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:47.755275 23909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:47.755430 23909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:47.755973 23909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:22:47.756980 23988 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 06:22:47.761968 23909 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 06:22:47.762043 23909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 06:22:47.762116 23909 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 06:22:47.763674 23909 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 06:22:47.763715 23909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 06:22:47.764019 23988 tablet_bootstrap.cc:492] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a: Bootstrap starting.
I20260810 06:22:47.772763 23909 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:43547
I20260810 06:22:47.772847 24171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:43547 every 8 connection(s)
I20260810 06:22:47.773648 23909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:22:47.776090 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 23909
I20260810 06:22:47.776549 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:37759
--local_ip_for_outbound_sockets=127.31.204.194
--tserver_master_addrs=127.31.204.254:46155
--webserver_port=36507
--webserver_interface=127.31.204.194
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:22:47.783702 23988 log.cc:824] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a: Log is configured to *not* fsync() on all Append() calls
Aug 10 06:22:47 dist-test-slave-q9h9 krb5kdc[22838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342967, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:22:47.791618 23988 tablet_bootstrap.cc:492] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:22:47.792048 23988 tablet_bootstrap.cc:492] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a: Bootstrap complete.
I20260810 06:22:47.792743 24177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.775734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:38189)
0810 06:22:47.776636 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:47.776693 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:47.777721 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:47.779299 (+ 1578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:47.779308 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:47.779670 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:47.780100 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.780112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.781984 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.781988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.783337 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.783349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.784279 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.784283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.784499 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.785134 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:47.785154 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:47.787040 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:47.789407 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:47.789413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:47.789426 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:47.789749 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:47.790158 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:47.790161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:47.790163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:47.790335 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:47.790967 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:47.790973 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:47.791245 (+ 272us) client_negotiation.cc:769] Sending connection context
0810 06:22:47.791429 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 06:22:47.791831 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":2,"thread_start_us":158,"threads_started":1}
I20260810 06:22:47.792800 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.775792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:38189 (local address 127.31.204.254:46155)
0810 06:22:47.776327 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:47.776346 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:47.778041 (+ 1695us) server_negotiation.cc:407] Connection header received
0810 06:22:47.778350 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:47.778364 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:47.778810 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:47.779214 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:47.780298 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.781813 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.783477 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.784141 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.784300 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.787249 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:47.787255 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:47.787271 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:47.787305 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:47.789280 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:47.789876 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:47.789881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:47.789887 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:47.790071 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:47.790427 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:47.790430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:47.790432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:47.790858 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:47.790990 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:47.791569 (+ 579us) server_negotiation.cc:299] Negotiation successful
0810 06:22:47.791954 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":132,"threads_started":1}
I20260810 06:22:47.793398 23988 ts_tablet_manager.cc:1397] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.004s
I20260810 06:22:47.795678 24172 heartbeater.cc:347] Connected to a master server at 127.31.204.254:46155
I20260810 06:22:47.795903 24172 heartbeater.cc:464] Registering TS with master...
I20260810 06:22:47.795934 23988 raft_consensus.cc:348] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } } peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } }
I20260810 06:22:47.796422 23988 raft_consensus.cc:729] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: befeb79e90b2436ea19b3a412dfe0c0a, State: Initialized, Role: FOLLOWER
I20260810 06:22:47.796422 24172 heartbeater.cc:511] Master 127.31.204.254:46155 requested a full tablet report, sending...
I20260810 06:22:47.796900 23988 consensus_queue.cc:260] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } } peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } }
I20260810 06:22:47.797856 23988 ts_tablet_manager.cc:1428] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20260810 06:22:47.799371 23842 ts_manager.cc:194] Registered new tserver with Master: befeb79e90b2436ea19b3a412dfe0c0a (127.31.204.193:43547)
I20260810 06:22:47.802529 23842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:38189
I20260810 06:22:47.804764 24172 heartbeater.cc:503] Master 127.31.204.254:46155 was elected leader, sending a full tablet report...
Aug 10 06:22:47 dist-test-slave-q9h9 krb5kdc[22838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342967, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for kudu/127.31.204.193@KRBTEST.COM
I20260810 06:22:47.812323 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.801936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:43547 (local address 127.0.0.1:38372)
0810 06:22:47.802675 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:47.802726 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:47.802855 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:47.803444 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:47.803449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:47.803854 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:47.803972 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.803979 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.805225 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.805229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.806128 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.806139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.806946 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.806948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.807026 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.807520 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:47.807530 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:47.808925 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:47.810827 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:47.810835 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:47.810839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:47.811046 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:47.811314 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:47.811316 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:47.811317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:47.811383 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:47.811835 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:47.811839 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:47.811926 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 06:22:47.812028 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 06:22:47.812204 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":129,"threads_started":1}
I20260810 06:22:47.812430 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.802108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38372 (local address 127.31.204.193:43547)
0810 06:22:47.802420 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:47.802431 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:47.802819 (+ 388us) server_negotiation.cc:407] Connection header received
0810 06:22:47.802981 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:47.802986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:47.803168 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:47.803288 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:47.804099 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.805113 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.806366 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.806845 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.806950 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.809090 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:47.809100 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:47.809111 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:47.809137 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:47.810719 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:47.811142 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:47.811147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:47.811149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:47.811239 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:47.811460 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:47.811463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:47.811465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:47.811736 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:47.811872 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:47.812120 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 06:22:47.812278 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":234,"thread_start_us":83,"threads_started":1}
I20260810 06:22:47.813339 24064 tablet_service.cc:1514] Processing DeleteTablet for tablet bb27a68d99da486fbc522654d963573d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:22:47 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:38372
I20260810 06:22:47.814102 24188 tablet_replica.cc:330] stopping tablet replica
I20260810 06:22:47.814468 24188 raft_consensus.cc:2229] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:22:47.814697 24188 raft_consensus.cc:2258] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:22:47.815582 24188 ts_tablet_manager.cc:1905] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:22:47.819131 24188 ts_tablet_manager.cc:1918] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:22:47.819212 24188 log.cc:1196] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/wal/wals/bb27a68d99da486fbc522654d963573d
I20260810 06:22:47.819449 24188 ts_tablet_manager.cc:1939] T bb27a68d99da486fbc522654d963573d P befeb79e90b2436ea19b3a412dfe0c0a: Deleting consensus metadata
I20260810 06:22:47.820886 23816 catalog_manager.cc:4841] TS befeb79e90b2436ea19b3a412dfe0c0a (127.31.204.193:43547): tablet bb27a68d99da486fbc522654d963573d (table test-table [id=6906bef31f124f24b78fa45ae35090a6]) successfully deleted
W20260810 06:22:47.890774 24179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:47.890995 24179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:47.891052 24179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:47.894977 24179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:47.895025 24179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:47.895099 24179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:22:47.899981 24179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:37759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=36507
--tserver_master_addrs=127.31.204.254:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:47.900848 24179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:47.901742 24179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:22:47.910470 24179 server_base.cc:1034] running on GCE node
W20260810 06:22:47.910437 24199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:47.910493 24203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:47.910436 24201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:47.911072 24179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:47.911633 24179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:47.912786 24179 hybrid_clock.cc:648] HybridClock initialized: now 1786342967912765 us; error 39 us; skew 500 ppm
Aug 10 06:22:47 dist-test-slave-q9h9 krb5kdc[22838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342967, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:47.915356 24179 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:22:47.916141 24179 webserver.cc:460] Webserver started at http://127.31.204.194:36507/ using document root <none> and password file <none>
I20260810 06:22:47.916618 24179 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:47.916709 24179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:47.920166 24179 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260810 06:22:47.922107 24216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:47.923509 24179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:22:47.923619 24179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "807c1bfaa46c4ed098402fdf225ed4c6"
format_stamp: "Formatted at 2026-08-10 06:22:46 on dist-test-slave-q9h9"
server_key: "ad67e6329f60a25f758907e4ebd25c9e"
server_key_iv: "391b85b05fa64f2ee2615e3d9b3b18d7"
server_key_version: "encryptionkey@0"
I20260810 06:22:47.924093 24179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:47.941267 24179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:47.944022 24179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:47.944245 24179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:47.944821 24179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:22:47.945824 24230 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 06:22:47.949497 24179 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 06:22:47.949548 24179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260810 06:22:47.949609 24179 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 06:22:47.951202 24179 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 06:22:47.951246 24179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260810 06:22:47.951480 24230 tablet_bootstrap.cc:492] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6: Bootstrap starting.
I20260810 06:22:47.956878 24230 log.cc:824] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6: Log is configured to *not* fsync() on all Append() calls
I20260810 06:22:47.960644 24179 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:37759
I20260810 06:22:47.960708 24399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:37759 every 8 connection(s)
I20260810 06:22:47.961412 24179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:22:47.963940 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 24179
I20260810 06:22:47.964172 24230 tablet_bootstrap.cc:492] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:22:47.964344 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:38687
--local_ip_for_outbound_sockets=127.31.204.195
--tserver_master_addrs=127.31.204.254:46155
--webserver_port=32953
--webserver_interface=127.31.204.195
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:22:47.964562 24230 tablet_bootstrap.cc:492] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6: Bootstrap complete.
I20260810 06:22:47.964908 24230 ts_tablet_manager.cc:1397] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260810 06:22:47.967373 24230 raft_consensus.cc:348] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } } peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } }
I20260810 06:22:47.967826 24230 raft_consensus.cc:729] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 807c1bfaa46c4ed098402fdf225ed4c6, State: Initialized, Role: FOLLOWER
I20260810 06:22:47.968245 24230 consensus_queue.cc:260] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } } peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } }
I20260810 06:22:47.969089 24230 ts_tablet_manager.cc:1428] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6: Time spent starting tablet: real 0.004s user 0.002s sys 0.004s
Aug 10 06:22:47 dist-test-slave-q9h9 krb5kdc[22838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342967, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:22:47.974691 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.963065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:59229 (local address 127.31.204.254:46155)
0810 06:22:47.963207 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:47.963212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:47.963758 (+ 546us) server_negotiation.cc:407] Connection header received
0810 06:22:47.964887 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:47.964890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:47.964950 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:47.965033 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:47.966505 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.967023 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.968142 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.968513 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.968626 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.971267 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:47.971271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:47.971274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:47.971304 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:47.972767 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:47.973211 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:47.973214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:47.973216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:47.973316 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:47.973598 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:47.973601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:47.973603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:47.973827 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:47.973931 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:47.974393 (+ 462us) server_negotiation.cc:299] Negotiation successful
0810 06:22:47.974529 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:22:47.975322 24403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.963151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:59229)
0810 06:22:47.963612 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:47.963663 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:47.964665 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:47.965297 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:47.965306 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:47.965732 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:47.966366 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.966381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.967161 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.967164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.967997 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.968008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.968626 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.968629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.968815 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.969664 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:47.969680 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:47.971087 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:47.972876 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:47.972881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:47.972890 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:47.973121 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:47.973401 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:47.973403 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:47.973404 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:47.973519 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:47.973914 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:47.973918 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:47.974170 (+ 252us) client_negotiation.cc:769] Sending connection context
0810 06:22:47.974317 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 06:22:47.974537 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":128,"threads_started":1}
I20260810 06:22:47.976380 24400 heartbeater.cc:347] Connected to a master server at 127.31.204.254:46155
I20260810 06:22:47.976614 24400 heartbeater.cc:464] Registering TS with master...
I20260810 06:22:47.977135 24400 heartbeater.cc:511] Master 127.31.204.254:46155 requested a full tablet report, sending...
I20260810 06:22:47.978969 23842 ts_manager.cc:194] Registered new tserver with Master: 807c1bfaa46c4ed098402fdf225ed4c6 (127.31.204.194:37759)
I20260810 06:22:47.980109 23842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:59229
I20260810 06:22:47.982283 24400 heartbeater.cc:503] Master 127.31.204.254:46155 was elected leader, sending a full tablet report...
Aug 10 06:22:47 dist-test-slave-q9h9 krb5kdc[22838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342967, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for kudu/127.31.204.194@KRBTEST.COM
I20260810 06:22:47.988586 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.979926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:37759 (local address 127.0.0.1:43402)
0810 06:22:47.980380 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:47.980394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:47.980464 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:47.980939 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:47.980942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:47.981115 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:47.981219 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.981227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.982648 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.982651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.983260 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:47.983270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.983917 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.983921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:47.984011 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.984396 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:47.984407 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:47.985512 (+ 1105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:47.987201 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:47.987204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:47.987206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:47.987389 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:47.987650 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:47.987653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:47.987655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:47.987740 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:47.988122 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:47.988125 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:47.988217 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 06:22:47.988304 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 06:22:47.988441 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384}
I20260810 06:22:47.988665 24409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:47.980037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43402 (local address 127.31.204.194:37759)
0810 06:22:47.980320 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:47.980331 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:47.980537 (+ 206us) server_negotiation.cc:407] Connection header received
0810 06:22:47.980595 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:47.980599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:47.980788 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:47.981362 (+ 574us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:47.981498 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.982533 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.983368 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:47.983813 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:47.983905 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:47.985635 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:47.985644 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:47.985656 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:47.985679 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:47.987100 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:47.987487 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:47.987493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:47.987495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:47.987568 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:47.987819 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:47.987823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:47.987824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:47.988024 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:47.988137 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:47.988394 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 06:22:47.988522 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
I20260810 06:22:47.989573 24293 tablet_service.cc:1514] Processing DeleteTablet for tablet bb27a68d99da486fbc522654d963573d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:22:47 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:43402
I20260810 06:22:47.990391 24412 tablet_replica.cc:330] stopping tablet replica
I20260810 06:22:47.990639 24412 raft_consensus.cc:2229] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:22:47.990840 24412 raft_consensus.cc:2258] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:22:47.991680 24412 ts_tablet_manager.cc:1905] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:22:47.995153 24412 ts_tablet_manager.cc:1918] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:22:47.995245 24412 log.cc:1196] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/bb27a68d99da486fbc522654d963573d
I20260810 06:22:47.995510 24412 ts_tablet_manager.cc:1939] T bb27a68d99da486fbc522654d963573d P 807c1bfaa46c4ed098402fdf225ed4c6: Deleting consensus metadata
I20260810 06:22:47.996809 23816 catalog_manager.cc:4841] TS 807c1bfaa46c4ed098402fdf225ed4c6 (127.31.204.194:37759): tablet bb27a68d99da486fbc522654d963573d (table test-table [id=6906bef31f124f24b78fa45ae35090a6]) successfully deleted
W20260810 06:22:48.080854 24405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:48.081066 24405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:48.081105 24405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:48.084915 24405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:48.084965 24405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:48.085017 24405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:22:48.089818 24405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:38687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=32953
--tserver_master_addrs=127.31.204.254:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:48.090720 24405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:48.091598 24405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:22:48.100605 24425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:48.100612 24419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:48.100612 24422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:48.100914 24405 server_base.cc:1034] running on GCE node
I20260810 06:22:48.101294 24405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:48.101871 24405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:48.103034 24405 hybrid_clock.cc:648] HybridClock initialized: now 1786342968103015 us; error 55 us; skew 500 ppm
Aug 10 06:22:48 dist-test-slave-q9h9 krb5kdc[22838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342968, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:48.105495 24405 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:22:48.106297 24405 webserver.cc:460] Webserver started at http://127.31.204.195:32953/ using document root <none> and password file <none>
I20260810 06:22:48.106837 24405 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:48.106932 24405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:48.110539 24405 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:22:48.112376 24439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:48.113534 24405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:22:48.113631 24405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "bac5e0a95e9442bfa498f54d0b98c598"
format_stamp: "Formatted at 2026-08-10 06:22:47 on dist-test-slave-q9h9"
server_key: "dff70594f592cc55ef9143b416fb6b18"
server_key_iv: "438bb2f9530a16cccacfb492a80a1319"
server_key_version: "encryptionkey@0"
I20260810 06:22:48.114153 24405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:48.125968 24405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:48.128623 24405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:48.128772 24405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:48.129292 24405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:22:48.130061 24405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:22:48.130116 24405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:48.130162 24405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:22:48.130191 24405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:48.138816 24405 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:38687
I20260810 06:22:48.138839 24626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:38687 every 8 connection(s)
I20260810 06:22:48.139696 24405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:22:48.147125 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 24405
Aug 10 06:22:48 dist-test-slave-q9h9 krb5kdc[22838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342968, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:22:48.152197 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:48.141191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:60799 (local address 127.31.204.254:46155)
0810 06:22:48.141309 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:48.141312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:48.142128 (+ 816us) server_negotiation.cc:407] Connection header received
0810 06:22:48.143366 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:48.143369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:48.143417 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:48.143487 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:48.144608 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.145129 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.146234 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.146445 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.146516 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:48.148813 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:48.148817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:48.148820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:48.148844 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:48.150163 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:48.150636 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:48.150640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:48.150642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:48.150728 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:48.151050 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:48.151054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:48.151056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:48.151281 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:48.151398 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:48.151857 (+ 459us) server_negotiation.cc:299] Negotiation successful
0810 06:22:48.152000 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 06:22:48.153074 24638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:48.141540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:60799)
0810 06:22:48.141964 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:48.142002 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:48.143182 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:48.143649 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:48.143656 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:48.143998 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:48.144472 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:48.144485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.145246 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.145249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:48.146147 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:48.146156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.146549 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.146551 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:48.146689 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:48.147178 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:48.147193 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:48.148668 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:48.150272 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:48.150279 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:48.150291 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:48.150542 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:48.150809 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:48.150830 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:48.150832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:48.150973 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:48.151388 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:48.151395 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:48.151634 (+ 239us) client_negotiation.cc:769] Sending connection context
0810 06:22:48.151793 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 06:22:48.152090 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":116,"threads_started":1}
I20260810 06:22:48.154242 24627 heartbeater.cc:347] Connected to a master server at 127.31.204.254:46155
I20260810 06:22:48.154482 24627 heartbeater.cc:464] Registering TS with master...
I20260810 06:22:48.155054 24641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:48.147757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39582 (local address 127.31.204.254:46155)
0810 06:22:48.148011 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:48.148015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:48.148121 (+ 106us) server_negotiation.cc:407] Connection header received
0810 06:22:48.148246 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:48.148249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:48.148298 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:48.148378 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:48.149066 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.149583 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.150488 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.150705 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.150789 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:48.152012 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:48.152016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:48.152018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:48.152040 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:48.153470 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:48.153849 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:48.153853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:48.153855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:48.153930 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:48.154209 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:48.154211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:48.154213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:48.154433 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:48.154630 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:48.154772 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 06:22:48.154904 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":87,"threads_started":1}
I20260810 06:22:48.155056 24627 heartbeater.cc:511] Master 127.31.204.254:46155 requested a full tablet report, sending...
I20260810 06:22:48.156140 23842 ts_manager.cc:194] Registered new tserver with Master: bac5e0a95e9442bfa498f54d0b98c598 (127.31.204.195:38687)
I20260810 06:22:48.156656 23842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:60799
I20260810 06:22:48.168051 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:22:48.171698 24641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:48.168606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39594 (local address 127.31.204.254:46155)
0810 06:22:48.168702 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:48.168705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:48.168868 (+ 163us) server_negotiation.cc:407] Connection header received
0810 06:22:48.168987 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:48.168989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:48.169070 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:48.169145 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:22:48.169336 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.169798 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.170591 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.170805 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.170874 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:48.171080 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:22:48.171418 (+ 338us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:22:48.171497 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:48.171563 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 06:22:48.171608 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 06:22:48.172245 23842 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:22:48.175365 23842 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:22:48.182336 24293 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad4f2aee14348ac9844cd86272080ae (DEFAULT_TABLE table=test-table [id=74d3882f306e464faf38295c43397007]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:22:48.182336 24064 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad4f2aee14348ac9844cd86272080ae (DEFAULT_TABLE table=test-table [id=74d3882f306e464faf38295c43397007]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:22:48.182950 24293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad4f2aee14348ac9844cd86272080ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:22:48.182950 24064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad4f2aee14348ac9844cd86272080ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:22:48.185417 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:48.181471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:38687 (local address 127.0.0.1:42146)
0810 06:22:48.181631 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:48.181644 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:48.181715 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:48.182380 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:48.182383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:48.182400 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:22:48.182481 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:48.182488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.183525 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.183528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:48.184260 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:48.184269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.185020 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.185024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:48.185117 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:48.185158 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 06:22:48.185229 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 06:22:48.185302 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 06:22:48.185720 24230 tablet_bootstrap.cc:492] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6: Bootstrap starting.
I20260810 06:22:48.185739 23988 tablet_bootstrap.cc:492] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a: Bootstrap starting.
I20260810 06:22:48.186102 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:48.181624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42146 (local address 127.31.204.195:38687)
0810 06:22:48.181956 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:48.181968 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:48.181989 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:22:48.182076 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:48.182082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:48.182242 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:48.182378 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:22:48.182581 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.183404 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.184373 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.184907 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.185016 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:48.185139 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:48.185847 (+ 708us) server_negotiation.cc:299] Negotiation successful
0810 06:22:48.185946 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":149,"threads_started":1}
I20260810 06:22:48.186646 23988 tablet_bootstrap.cc:654] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a: Neither blocks nor log segments found. Creating new log.
I20260810 06:22:48.186646 24230 tablet_bootstrap.cc:654] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6: Neither blocks nor log segments found. Creating new log.
I20260810 06:22:48.188089 23988 tablet_bootstrap.cc:492] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a: No bootstrap required, opened a new log
I20260810 06:22:48.188174 23988 ts_tablet_manager.cc:1397] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 06:22:48.188225 24230 tablet_bootstrap.cc:492] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6: No bootstrap required, opened a new log
I20260810 06:22:48.188318 24230 ts_tablet_manager.cc:1397] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 06:22:48.188566 23988 raft_consensus.cc:348] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } } peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } }
I20260810 06:22:48.188660 23988 raft_consensus.cc:374] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:22:48.188696 23988 raft_consensus.cc:729] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: befeb79e90b2436ea19b3a412dfe0c0a, State: Initialized, Role: FOLLOWER
I20260810 06:22:48.188737 24230 raft_consensus.cc:348] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } } peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } }
I20260810 06:22:48.188823 23988 consensus_queue.cc:260] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } } peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } }
I20260810 06:22:48.188867 24230 raft_consensus.cc:374] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:22:48.188905 24230 raft_consensus.cc:729] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 807c1bfaa46c4ed098402fdf225ed4c6, State: Initialized, Role: FOLLOWER
I20260810 06:22:48.189016 24230 consensus_queue.cc:260] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } } peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } }
I20260810 06:22:48.189062 23988 ts_tablet_manager.cc:1428] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:22:48.189287 24230 ts_tablet_manager.cc:1428] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:22:48.189177 24515 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad4f2aee14348ac9844cd86272080ae (DEFAULT_TABLE table=test-table [id=74d3882f306e464faf38295c43397007]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:22:48.190310 24515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad4f2aee14348ac9844cd86272080ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:22:48.196931 24658 tablet_bootstrap.cc:492] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598: Bootstrap starting.
I20260810 06:22:48.199306 24658 tablet_bootstrap.cc:654] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598: Neither blocks nor log segments found. Creating new log.
I20260810 06:22:48.199999 24658 log.cc:824] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598: Log is configured to *not* fsync() on all Append() calls
I20260810 06:22:48.201722 24658 tablet_bootstrap.cc:492] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598: No bootstrap required, opened a new log
I20260810 06:22:48.201901 24658 ts_tablet_manager.cc:1397] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:22:48.204744 24658 raft_consensus.cc:348] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } } peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } }
I20260810 06:22:48.204950 24658 raft_consensus.cc:374] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:22:48.205005 24658 raft_consensus.cc:729] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac5e0a95e9442bfa498f54d0b98c598, State: Initialized, Role: FOLLOWER
I20260810 06:22:48.205370 24658 consensus_queue.cc:260] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } } peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } }
I20260810 06:22:48.205927 24627 heartbeater.cc:503] Master 127.31.204.254:46155 was elected leader, sending a full tablet report...
I20260810 06:22:48.206281 24658 ts_tablet_manager.cc:1428] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260810 06:22:48.212296 24401 tablet.cc:2369] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:22:48.274771 24174 tablet.cc:2369] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:22:48.303493 24660 raft_consensus.cc:482] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:22:48.303699 24660 raft_consensus.cc:504] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } } peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } }
I20260810 06:22:48.304774 24660 leader_election.cc:288] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 807c1bfaa46c4ed098402fdf225ed4c6 (127.31.204.194:37759), befeb79e90b2436ea19b3a412dfe0c0a (127.31.204.193:43547)
I20260810 06:22:48.309114 24638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:48.305008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:37759 (local address 127.31.204.195:54061)
0810 06:22:48.305117 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:48.305129 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:48.305219 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:48.305582 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:48.305585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:48.305606 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:22:48.305744 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:48.305751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.306737 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.306739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:48.307897 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:48.307909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.308757 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.308760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:48.308841 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:48.308862 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 06:22:48.308911 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 06:22:48.308984 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 06:22:48.309151 24409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:48.305104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:54061 (local address 127.31.204.194:37759)
0810 06:22:48.305234 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:48.305239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:48.305268 (+ 29us) server_negotiation.cc:407] Connection header received
0810 06:22:48.305326 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:48.305328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:48.305390 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:48.305474 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:22:48.305854 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.306641 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.308024 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.308641 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.308771 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:48.308926 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:48.308964 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 06:22:48.309032 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 06:22:48.309355 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:48.305413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:41027 (local address 127.31.204.193:43547)
0810 06:22:48.305566 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:48.305571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:48.305694 (+ 123us) server_negotiation.cc:407] Connection header received
0810 06:22:48.305793 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:48.305796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:48.305861 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:48.305951 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:22:48.306224 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.306882 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.308201 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.308857 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.308971 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:48.309090 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:48.309169 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 06:22:48.309235 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 06:22:48.309533 24661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:48.305288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:43547 (local address 127.31.204.195:41027)
0810 06:22:48.305608 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:48.305621 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:48.305717 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:48.305972 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:48.305974 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:48.305990 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:22:48.306127 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:48.306132 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.306973 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.306975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:48.308096 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:48.308104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.308964 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.308968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:48.309052 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:48.309072 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 06:22:48.309154 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 06:22:48.309358 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":120,"threads_started":1}
I20260810 06:22:48.309712 24326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad4f2aee14348ac9844cd86272080ae" candidate_uuid: "bac5e0a95e9442bfa498f54d0b98c598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" is_pre_election: true
I20260810 06:22:48.310045 24326 raft_consensus.cc:2454] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bac5e0a95e9442bfa498f54d0b98c598 in term 0.
I20260810 06:22:48.310132 24103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad4f2aee14348ac9844cd86272080ae" candidate_uuid: "bac5e0a95e9442bfa498f54d0b98c598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" is_pre_election: true
I20260810 06:22:48.310420 24103 raft_consensus.cc:2454] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bac5e0a95e9442bfa498f54d0b98c598 in term 0.
I20260810 06:22:48.310501 24443 leader_election.cc:302] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 807c1bfaa46c4ed098402fdf225ed4c6, bac5e0a95e9442bfa498f54d0b98c598; no voters:
I20260810 06:22:48.310770 24660 raft_consensus.cc:2781] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:22:48.310851 24660 raft_consensus.cc:482] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:22:48.310901 24660 raft_consensus.cc:3037] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:22:48.311975 24660 raft_consensus.cc:504] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } } peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } }
I20260810 06:22:48.312305 24660 leader_election.cc:288] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [CANDIDATE]: Term 1 election: Requested vote from peers 807c1bfaa46c4ed098402fdf225ed4c6 (127.31.204.194:37759), befeb79e90b2436ea19b3a412dfe0c0a (127.31.204.193:43547)
I20260810 06:22:48.312664 24103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad4f2aee14348ac9844cd86272080ae" candidate_uuid: "bac5e0a95e9442bfa498f54d0b98c598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "befeb79e90b2436ea19b3a412dfe0c0a"
I20260810 06:22:48.312793 24103 raft_consensus.cc:3037] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:22:48.313277 24326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad4f2aee14348ac9844cd86272080ae" candidate_uuid: "bac5e0a95e9442bfa498f54d0b98c598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "807c1bfaa46c4ed098402fdf225ed4c6"
I20260810 06:22:48.313406 24326 raft_consensus.cc:3037] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:22:48.313943 24103 raft_consensus.cc:2454] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bac5e0a95e9442bfa498f54d0b98c598 in term 1.
I20260810 06:22:48.314221 24443 leader_election.cc:302] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bac5e0a95e9442bfa498f54d0b98c598, befeb79e90b2436ea19b3a412dfe0c0a; no voters:
I20260810 06:22:48.314414 24660 raft_consensus.cc:2781] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:22:48.314617 24660 raft_consensus.cc:686] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [term 1 LEADER]: Becoming Leader. State: Replica: bac5e0a95e9442bfa498f54d0b98c598, State: Running, Role: LEADER
I20260810 06:22:48.314654 24326 raft_consensus.cc:2454] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bac5e0a95e9442bfa498f54d0b98c598 in term 1.
I20260810 06:22:48.314958 24660 consensus_queue.cc:237] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } } peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } }
I20260810 06:22:48.318131 23838 catalog_manager.cc:5495] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 reported cstate change: term changed from 0 to 1, leader changed from <none> to bac5e0a95e9442bfa498f54d0b98c598 (127.31.204.195). New cstate: current_term: 1 leader_uuid: "bac5e0a95e9442bfa498f54d0b98c598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } health_report { overall_health: UNKNOWN } } }
I20260810 06:22:48.340306 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:48.337251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42160 (local address 127.31.204.195:38687)
0810 06:22:48.337380 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:48.337383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:48.337397 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:22:48.337508 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:48.337510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:48.337565 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:48.337646 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:22:48.337823 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.338357 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.339099 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:48.339297 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:48.339359 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:48.339584 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:22:48.339991 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:22:48.340076 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:48.340103 (+ 27us) server_negotiation.cc:299] Negotiation successful
0810 06:22:48.340151 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:22:48.345994 24326 raft_consensus.cc:1261] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6 [term 1 FOLLOWER]: Refusing update from remote peer bac5e0a95e9442bfa498f54d0b98c598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:22:48.345994 24103 raft_consensus.cc:1261] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Refusing update from remote peer bac5e0a95e9442bfa498f54d0b98c598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:22:48.346652 24660 consensus_queue.cc:1035] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:22:48.346849 24666 consensus_queue.cc:1035] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:22:48.351853 24668 mvcc.cc:205] Tried to move back new op lower bound from 7316860798338478080 to 7316860798220873728. Current Snapshot: MvccSnapshot[applied={T|T < 7316860798338478080}]
I20260810 06:22:48.352349 24672 mvcc.cc:205] Tried to move back new op lower bound from 7316860798338478080 to 7316860798220873728. Current Snapshot: MvccSnapshot[applied={T|T < 7316860798338478080}]
I20260810 06:22:48.352720 24669 mvcc.cc:205] Tried to move back new op lower bound from 7316860798338478080 to 7316860798220873728. Current Snapshot: MvccSnapshot[applied={T|T < 7316860798338478080}]
I20260810 06:22:48.360180 23838 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:22:48.360328 23838 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:22:48.362447 23838 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c: Sending DeleteTablet for 3 replicas of tablet 3ad4f2aee14348ac9844cd86272080ae
I20260810 06:22:48.363093 24064 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ad4f2aee14348ac9844cd86272080ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:22:48 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:38372
I20260810 06:22:48.363109 24515 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ad4f2aee14348ac9844cd86272080ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:22:48 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:42146
I20260810 06:22:48.363421 24293 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ad4f2aee14348ac9844cd86272080ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:22:48 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:43402
I20260810 06:22:48.363461 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 23909
I20260810 06:22:48.363579 24412 tablet_replica.cc:330] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6: stopping tablet replica
I20260810 06:22:48.363682 24412 raft_consensus.cc:2229] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:22:48.363791 24412 raft_consensus.cc:2258] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:22:48.375562 24412 ts_tablet_manager.cc:1905] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:22:48.388808 24412 ts_tablet_manager.cc:1918] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:22:48.388952 24412 log.cc:1196] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/3ad4f2aee14348ac9844cd86272080ae
I20260810 06:22:48.389333 24412 ts_tablet_manager.cc:1939] T 3ad4f2aee14348ac9844cd86272080ae P 807c1bfaa46c4ed098402fdf225ed4c6: Deleting consensus metadata
I20260810 06:22:48.390425 23816 catalog_manager.cc:4841] TS 807c1bfaa46c4ed098402fdf225ed4c6 (127.31.204.194:37759): tablet 3ad4f2aee14348ac9844cd86272080ae (table test-table [id=74d3882f306e464faf38295c43397007]) successfully deleted
I20260810 06:22:48.398334 24687 tablet_replica.cc:330] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598: stopping tablet replica
I20260810 06:22:48.398931 24687 raft_consensus.cc:2229] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:22:48.399411 24687 raft_consensus.cc:2258] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:22:48.400914 24687 ts_tablet_manager.cc:1905] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:22:48.403748 24687 ts_tablet_manager.cc:1918] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:22:48.403863 24687 log.cc:1196] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/3ad4f2aee14348ac9844cd86272080ae
I20260810 06:22:48.404176 24687 ts_tablet_manager.cc:1939] T 3ad4f2aee14348ac9844cd86272080ae P bac5e0a95e9442bfa498f54d0b98c598: Deleting consensus metadata
I20260810 06:22:48.405493 23815 catalog_manager.cc:4841] TS bac5e0a95e9442bfa498f54d0b98c598 (127.31.204.195:38687): tablet 3ad4f2aee14348ac9844cd86272080ae (table test-table [id=74d3882f306e464faf38295c43397007]) successfully deleted
W20260810 06:22:48.405946 23816 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.204.193:43547 (error 108)
I20260810 06:22:48.406176 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 24179
I20260810 06:22:48.409819 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:48.409401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:43547 (local address 127.0.0.1:39890)
0810 06:22:48.409622 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:48.409717 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260810 06:22:48.410233 23816 catalog_manager.cc:4584] TS befeb79e90b2436ea19b3a412dfe0c0a (127.31.204.193:43547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3ad4f2aee14348ac9844cd86272080ae: Network error: Client connection negotiation failed: client connection to 127.31.204.193:43547: connect: Connection refused (error 111)
I20260810 06:22:48.417035 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 24405
I20260810 06:22:48.428795 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 23775
I20260810 06:22:48.438145 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:46155
--webserver_interface=127.31.204.254
--webserver_port=41155
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:46155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:22:48.545992 24690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:48.546247 24690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:48.546295 24690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:48.550159 24690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:22:48.550202 24690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:48.550220 24690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:48.550240 24690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:22:48.550275 24690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:22:48.555267 24690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:46155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:46155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=41155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:48.556195 24690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:48.557164 24690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:22:48.565045 24700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:48.565043 24704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:48.565135 24699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:48.565303 24690 server_base.cc:1034] running on GCE node
I20260810 06:22:48.565883 24690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:48.566917 24690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:48.568073 24690 hybrid_clock.cc:648] HybridClock initialized: now 1786342968568053 us; error 31 us; skew 500 ppm
Aug 10 06:22:48 dist-test-slave-q9h9 krb5kdc[22838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342968, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:48.570585 24690 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:22:48.571373 24690 webserver.cc:460] Webserver started at http://127.31.204.254:41155/ using document root <none> and password file <none>
I20260810 06:22:48.571878 24690 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:48.571945 24690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:48.575511 24690 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:22:48.577435 24721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:48.578541 24690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:22:48.578649 24690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "e0071217f0dd496e8982e7f70b22868c"
format_stamp: "Formatted at 2026-08-10 06:22:46 on dist-test-slave-q9h9"
server_key: "7880668273f7bdf91310b09fa8715431"
server_key_iv: "501d280993ecb494b5310ec25761e9eb"
server_key_version: "encryptionkey@0"
I20260810 06:22:48.579142 24690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:48.612531 24690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:48.615306 24690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:48.615504 24690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:48.623344 24690 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:46155
I20260810 06:22:48.623605 24813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:46155 every 8 connection(s)
I20260810 06:22:48.624478 24690 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:22:48.628715 24816 sys_catalog.cc:263] Verifying existing consensus state
I20260810 06:22:48.630364 24816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c: Bootstrap starting.
I20260810 06:22:48.634135 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 24690
I20260810 06:22:48.634635 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:43547
--local_ip_for_outbound_sockets=127.31.204.193
--tserver_master_addrs=127.31.204.254:46155
--webserver_port=41079
--webserver_interface=127.31.204.193
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:22:48.648821 24816 log.cc:824] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c: Log is configured to *not* fsync() on all Append() calls
I20260810 06:22:48.659853 24816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:22:48.660169 24816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c: Bootstrap complete.
I20260810 06:22:48.663975 24816 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0071217f0dd496e8982e7f70b22868c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46155 } }
I20260810 06:22:48.664338 24816 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0071217f0dd496e8982e7f70b22868c, State: Initialized, Role: FOLLOWER
I20260810 06:22:48.664801 24816 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0071217f0dd496e8982e7f70b22868c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46155 } }
I20260810 06:22:48.664921 24816 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:22:48.664984 24816 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:22:48.665071 24816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 2 FOLLOWER]: Advancing to term 3
I20260810 06:22:48.666874 24816 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0071217f0dd496e8982e7f70b22868c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46155 } }
I20260810 06:22:48.667162 24816 leader_election.cc:302] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0071217f0dd496e8982e7f70b22868c; no voters:
I20260810 06:22:48.667470 24816 leader_election.cc:288] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 06:22:48.667562 24824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 3 FOLLOWER]: Leader election won for term 3
I20260810 06:22:48.667763 24824 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [term 3 LEADER]: Becoming Leader. State: Replica: e0071217f0dd496e8982e7f70b22868c, State: Running, Role: LEADER
I20260810 06:22:48.668049 24824 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0071217f0dd496e8982e7f70b22868c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46155 } }
I20260810 06:22:48.668630 24816 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:22:48.668947 24824 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0071217f0dd496e8982e7f70b22868c. Latest consensus state: current_term: 3 leader_uuid: "e0071217f0dd496e8982e7f70b22868c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0071217f0dd496e8982e7f70b22868c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46155 } } }
I20260810 06:22:48.669140 24824 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [sys.catalog]: This master's current role is: LEADER
I20260810 06:22:48.669775 24825 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e0071217f0dd496e8982e7f70b22868c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0071217f0dd496e8982e7f70b22868c" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46155 } } }
I20260810 06:22:48.669934 24825 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0071217f0dd496e8982e7f70b22868c [sys.catalog]: This master's current role is: LEADER
I20260810 06:22:48.670511 24830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:22:48.674348 24830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:22:48.674862 24830 catalog_manager.cc:1259] Loaded cluster ID: 313efae9ae0f46ccbe9515d652c6af58
I20260810 06:22:48.674911 24830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 06:22:48.675784 24830 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa1f92c89ad google::LogMessage::Fail() at ??:0
@ 0x7fa1f92cb88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa1f92c84de google::LogMessage::Flush() at ??:0
@ 0x7fa1f92c9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa203c0bfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fa203c0cd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa203c09579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fa203c4d640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa1fb51a3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa1fb51ac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fa1fb51c3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa1fb50b512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa1fcb4f6db start_thread at ??:0
@ 0x7fa1f7dc661f clone at ??:0
W20260810 06:22:48.758126 24821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:48.758351 24821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:48.758409 24821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:48.762348 24821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:48.762401 24821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:48.762470 24821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:22:48.767529 24821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:43547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=41079
--tserver_master_addrs=127.31.204.254:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:48.768501 24821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:48.769481 24821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:22:48.778662 24912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:48.778702 24909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:48.778667 24907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:48.779045 24821 server_base.cc:1034] running on GCE node
I20260810 06:22:48.779490 24821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:48.780234 24821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:48.781433 24821 hybrid_clock.cc:648] HybridClock initialized: now 1786342968781407 us; error 32 us; skew 500 ppm
Aug 10 06:22:48 dist-test-slave-q9h9 krb5kdc[22838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342968, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:48.784608 24821 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:22:48.785657 24821 webserver.cc:460] Webserver started at http://127.31.204.193:41079/ using document root <none> and password file <none>
I20260810 06:22:48.786392 24821 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:48.786485 24821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:48.791370 24821 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:22:48.793962 24926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:48.795215 24821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:22:48.795307 24821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "befeb79e90b2436ea19b3a412dfe0c0a"
format_stamp: "Formatted at 2026-08-10 06:22:46 on dist-test-slave-q9h9"
server_key: "1dd3b43790a84bfbcadf562f8869396c"
server_key_iv: "166f7ee6c33de3a785ed77424662179b"
server_key_version: "encryptionkey@0"
I20260810 06:22:48.795886 24821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:48.804462 24821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:48.807299 24821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:48.807454 24821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:48.807982 24821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:22:48.808993 24938 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 06:22:48.814095 24821 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 06:22:48.814150 24821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 06:22:48.814217 24821 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 06:22:48.815778 24821 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 06:22:48.815821 24821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 06:22:48.816136 24938 tablet_bootstrap.cc:492] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a: Bootstrap starting.
I20260810 06:22:48.825521 24821 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:43547
I20260810 06:22:48.825601 25112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:43547 every 8 connection(s)
I20260810 06:22:48.826469 24821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:22:48.831939 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 24821
I20260810 06:22:48.832326 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:37759
--local_ip_for_outbound_sockets=127.31.204.194
--tserver_master_addrs=127.31.204.254:46155
--webserver_port=36507
--webserver_interface=127.31.204.194
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:22:48.837610 24938 log.cc:824] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a: Log is configured to *not* fsync() on all Append() calls
I20260810 06:22:48.846346 24938 tablet_bootstrap.cc:492] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:22:48.846741 24938 tablet_bootstrap.cc:492] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a: Bootstrap complete.
I20260810 06:22:48.848095 24938 ts_tablet_manager.cc:1397] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a: Time spent bootstrapping tablet: real 0.032s user 0.028s sys 0.001s
I20260810 06:22:48.851176 24938 raft_consensus.cc:348] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } } peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } }
I20260810 06:22:48.851686 24938 raft_consensus.cc:729] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: befeb79e90b2436ea19b3a412dfe0c0a, State: Initialized, Role: FOLLOWER
I20260810 06:22:48.852195 24938 consensus_queue.cc:260] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } } peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } }
I20260810 06:22:48.853224 24938 ts_tablet_manager.cc:1428] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260810 06:22:48.888949 25116 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:22:48.828281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:55407)
0810 06:22:48.828763 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:48.828816 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:48.829574 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:48.888035 (+ 58461us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":119,"threads_started":1}
W20260810 06:22:48.889519 25113 heartbeater.cc:644] Failed to heartbeat to 127.31.204.254:46155 (0 consecutive failures): Network error: Failed to ping master at 127.31.204.254:46155: Client connection negotiation failed: client connection to 127.31.204.254:46155: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 06:22:48.890264 25116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:48.889977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:42689)
0810 06:22:48.890126 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:48.890199 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 06:22:48.890892 25116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:48.890690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:38805)
0810 06:22:48.890787 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:48.890836 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 06:22:48.891047 25113 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 06:22:48.943739 25120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:48.943940 25120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:48.943979 25120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:48.947763 25120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:48.947804 25120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:48.947856 25120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:22:48.952608 25120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:37759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=36507
--tserver_master_addrs=127.31.204.254:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:48.953403 25120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:48.954254 25120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:22:48.962929 25135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:48.962929 25132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:48.962982 25129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:48.963002 25120 server_base.cc:1034] running on GCE node
I20260810 06:22:48.963536 25120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:48.964072 25120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:48.965225 25120 hybrid_clock.cc:648] HybridClock initialized: now 1786342968965209 us; error 31 us; skew 500 ppm
Aug 10 06:22:48 dist-test-slave-q9h9 krb5kdc[22838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342968, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:48.967697 25120 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:22:48.968436 25120 webserver.cc:460] Webserver started at http://127.31.204.194:36507/ using document root <none> and password file <none>
I20260810 06:22:48.968911 25120 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:48.968974 25120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:48.972391 25120 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:22:48.974213 25149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:48.975407 25120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:22:48.975517 25120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "807c1bfaa46c4ed098402fdf225ed4c6"
format_stamp: "Formatted at 2026-08-10 06:22:46 on dist-test-slave-q9h9"
server_key: "ad67e6329f60a25f758907e4ebd25c9e"
server_key_iv: "391b85b05fa64f2ee2615e3d9b3b18d7"
server_key_version: "encryptionkey@0"
I20260810 06:22:48.975978 25120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:48.985441 25120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:48.988085 25120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:48.988230 25120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:48.988720 25120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:22:48.989425 25120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:22:48.989470 25120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:48.989518 25120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:22:48.989544 25120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:48.997709 25120 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:37759
I20260810 06:22:48.997751 25328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:37759 every 8 connection(s)
I20260810 06:22:48.998633 25120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:22:49.001497 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 25120
I20260810 06:22:49.001857 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:38687
--local_ip_for_outbound_sockets=127.31.204.195
--tserver_master_addrs=127.31.204.254:46155
--webserver_port=32953
--webserver_interface=127.31.204.195
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:22:49.001889 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:49.000446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:43805)
0810 06:22:49.000859 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:49.001228 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":110,"threads_started":1}
W20260810 06:22:49.002308 25330 heartbeater.cc:644] Failed to heartbeat to 127.31.204.254:46155 (0 consecutive failures): Network error: Failed to ping master at 127.31.204.254:46155: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
I20260810 06:22:49.002820 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:49.002637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:50085)
0810 06:22:49.002727 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:49.002771 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 06:22:49.003340 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:49.003167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:43111)
0810 06:22:49.003248 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:49.003293 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 06:22:49.003504 25330 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 06:22:49.109035 25335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:22:49.109304 25335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:22:49.109359 25335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:22:49.113319 25335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:22:49.113371 25335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:22:49.113440 25335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:22:49.118721 25335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:38687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=32953
--tserver_master_addrs=127.31.204.254:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:22:49.119668 25335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:22:49.120633 25335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:22:49.129498 25346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:49.129519 25340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:22:49.129519 25348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:22:49.129823 25335 server_base.cc:1034] running on GCE node
I20260810 06:22:49.130184 25335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:22:49.130710 25335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:22:49.131857 25335 hybrid_clock.cc:648] HybridClock initialized: now 1786342969131833 us; error 37 us; skew 500 ppm
Aug 10 06:22:49 dist-test-slave-q9h9 krb5kdc[22838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342969, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:22:49.134301 25335 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:22:49.135106 25335 webserver.cc:460] Webserver started at http://127.31.204.195:32953/ using document root <none> and password file <none>
I20260810 06:22:49.135586 25335 fs_manager.cc:359] Metadata directory not provided
I20260810 06:22:49.135648 25335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:22:49.139132 25335 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:22:49.140969 25358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:49.142134 25335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:22:49.142230 25335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "bac5e0a95e9442bfa498f54d0b98c598"
format_stamp: "Formatted at 2026-08-10 06:22:47 on dist-test-slave-q9h9"
server_key: "dff70594f592cc55ef9143b416fb6b18"
server_key_iv: "438bb2f9530a16cccacfb492a80a1319"
server_key_version: "encryptionkey@0"
I20260810 06:22:49.142726 25335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:22:49.181865 25335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:22:49.184435 25335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:22:49.184599 25335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:22:49.185173 25335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:22:49.186081 25335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:22:49.186136 25335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:49.186180 25335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:22:49.186208 25335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:22:49.194496 25335 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:38687
I20260810 06:22:49.194543 25537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:38687 every 8 connection(s)
I20260810 06:22:49.195485 25335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:22:49.198235 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 25335
I20260810 06:22:49.199198 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:49.197451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:48273)
0810 06:22:49.197901 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:49.198374 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":104,"threads_started":1}
W20260810 06:22:49.199677 25539 heartbeater.cc:644] Failed to heartbeat to 127.31.204.254:46155 (0 consecutive failures): Network error: Failed to ping master at 127.31.204.254:46155: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
I20260810 06:22:49.200218 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:49.200001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:35397)
0810 06:22:49.200116 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:49.200164 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 06:22:49.200722 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:49.200567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:50767)
0810 06:22:49.200641 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:49.200678 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 06:22:49.200878 25539 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 06:22:49.891970 25544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:49.891563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:44655)
0810 06:22:49.891797 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:49.891853 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260810 06:22:50.004346 25546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:50.003972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:59285)
0810 06:22:50.004208 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:50.004257 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":66,"threads_started":1}
I20260810 06:22:50.201715 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:50.201344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:48335)
0810 06:22:50.201569 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:50.201620 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":67,"threads_started":1}
I20260810 06:22:50.663599 25548 raft_consensus.cc:482] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:22:50.663792 25548 raft_consensus.cc:504] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } } peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } }
I20260810 06:22:50.664830 25548 leader_election.cc:288] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bac5e0a95e9442bfa498f54d0b98c598 (127.31.204.195:38687), 807c1bfaa46c4ed098402fdf225ed4c6 (127.31.204.194:37759)
Aug 10 06:22:50 dist-test-slave-q9h9 krb5kdc[22838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342968, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.194@KRBTEST.COM
Aug 10 06:22:50 dist-test-slave-q9h9 krb5kdc[22838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342968, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.195@KRBTEST.COM
I20260810 06:22:50.677831 25554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:50.665039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:38687 (local address 127.31.204.193:33659)
0810 06:22:50.665298 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:50.665312 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:50.665416 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:50.667139 (+ 1723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:50.667146 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:50.667710 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:50.668198 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:50.668213 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:50.669451 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:50.669454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:50.670511 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:50.670521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:50.671189 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:50.671192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:50.671429 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:50.672003 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:50.672020 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:50.673793 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:50.675609 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:50.675612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:50.675613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:50.675952 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:50.676275 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:50.676278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:50.676280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:50.676436 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:50.677009 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:50.677019 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:50.677237 (+ 218us) client_negotiation.cc:769] Sending connection context
0810 06:22:50.677400 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 06:22:50.677632 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":185,"spinlock_wait_cycles":13952,"thread_start_us":73,"threads_started":1}
I20260810 06:22:50.678094 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:50.665050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:37759 (local address 127.31.204.193:43993)
0810 06:22:50.665428 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:50.665442 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:22:50.665552 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:22:50.667139 (+ 1587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:22:50.667146 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:22:50.667722 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:22:50.668199 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:50.668213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:50.669455 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:50.669458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:50.670513 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:22:50.670524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:22:50.671177 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:22:50.671180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:22:50.671429 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:50.672052 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:22:50.672064 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:22:50.673430 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:22:50.675551 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:50.675557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:50.675572 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:50.675955 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:50.676267 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:22:50.676270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:22:50.676273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:22:50.676437 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:22:50.677006 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:22:50.677012 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:22:50.677231 (+ 219us) client_negotiation.cc:769] Sending connection context
0810 06:22:50.677680 (+ 449us) client_negotiation.cc:240] Negotiation successful
0810 06:22:50.677908 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":174,"spinlock_wait_cycles":14976,"thread_start_us":77,"threads_started":1}
I20260810 06:22:50.678433 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:50.665325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:43993 (local address 127.31.204.194:37759)
0810 06:22:50.665645 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:50.665657 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:50.665691 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:22:50.666380 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:50.666388 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:50.666750 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:50.667030 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:50.668358 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:50.669315 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:50.670626 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:50.671062 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:50.671200 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:50.673668 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:50.673674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:50.673689 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:50.673723 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:50.675435 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:50.676060 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:50.676065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:50.676069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:50.676175 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:50.676526 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:50.676529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:50.676531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:50.676892 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:50.677006 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:50.677537 (+ 531us) server_negotiation.cc:299] Negotiation successful
0810 06:22:50.677836 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":62,"threads_started":1}
I20260810 06:22:50.678627 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:50.665213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:33659 (local address 127.31.204.195:38687)
0810 06:22:50.665637 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0810 06:22:50.665659 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 06:22:50.665691 (+ 32us) server_negotiation.cc:407] Connection header received
0810 06:22:50.666383 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:22:50.666389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:22:50.666750 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:22:50.667448 (+ 698us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:22:50.668365 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:50.669311 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:50.670628 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:22:50.671072 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:22:50.671221 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:22:50.673973 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:22:50.673978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:22:50.673988 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:22:50.674041 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:22:50.675501 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:50.676059 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:50.676064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:50.676068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:50.676182 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:22:50.676528 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:22:50.676531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:22:50.676533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:22:50.676897 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:22:50.677013 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 06:22:50.678214 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0810 06:22:50.678461 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":78,"threads_started":1}
W20260810 06:22:50.680620 24934 leader_election.cc:341] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 807c1bfaa46c4ed098402fdf225ed4c6 (127.31.204.194:37759): Not found: Tablet not found: 3ad4f2aee14348ac9844cd86272080ae
W20260810 06:22:50.680953 24931 leader_election.cc:341] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bac5e0a95e9442bfa498f54d0b98c598 (127.31.204.195:38687): Not found: Tablet not found: 3ad4f2aee14348ac9844cd86272080ae
I20260810 06:22:50.681042 24931 leader_election.cc:302] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: befeb79e90b2436ea19b3a412dfe0c0a; no voters: 807c1bfaa46c4ed098402fdf225ed4c6, bac5e0a95e9442bfa498f54d0b98c598
I20260810 06:22:50.681293 25548 raft_consensus.cc:2726] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 06:22:50.892889 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:50.892619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:44335)
0810 06:22:50.892747 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:50.892802 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 06:22:51.005375 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:51.004960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:34895)
0810 06:22:51.005214 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:51.005268 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
I20260810 06:22:51.202720 25561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:51.202338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:59683)
0810 06:22:51.202579 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:51.202634 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260810 06:22:51.893959 25562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:51.893550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:36811)
0810 06:22:51.893805 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:51.893860 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260810 06:22:52.006443 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:52.006007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:35423)
0810 06:22:52.006303 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:52.006355 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":89,"threads_started":1}
I20260810 06:22:52.203794 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:52.203377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:49425)
0810 06:22:52.203642 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:52.203702 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":86,"threads_started":1}
I20260810 06:22:52.379542 25566 raft_consensus.cc:482] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:22:52.379689 25566 raft_consensus.cc:504] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } } peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } }
I20260810 06:22:52.380090 25566 leader_election.cc:288] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bac5e0a95e9442bfa498f54d0b98c598 (127.31.204.195:38687), 807c1bfaa46c4ed098402fdf225ed4c6 (127.31.204.194:37759)
W20260810 06:22:52.380816 24931 leader_election.cc:341] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bac5e0a95e9442bfa498f54d0b98c598 (127.31.204.195:38687): Not found: Tablet not found: 3ad4f2aee14348ac9844cd86272080ae
W20260810 06:22:52.380923 24934 leader_election.cc:341] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 807c1bfaa46c4ed098402fdf225ed4c6 (127.31.204.194:37759): Not found: Tablet not found: 3ad4f2aee14348ac9844cd86272080ae
I20260810 06:22:52.381011 24934 leader_election.cc:302] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: befeb79e90b2436ea19b3a412dfe0c0a; no voters: 807c1bfaa46c4ed098402fdf225ed4c6, bac5e0a95e9442bfa498f54d0b98c598
I20260810 06:22:52.381147 25566 raft_consensus.cc:2726] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 06:22:52.895097 25567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:52.894669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:59827)
0810 06:22:52.894929 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:52.894990 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
I20260810 06:22:53.007474 25570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:53.007085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:59687)
0810 06:22:53.007320 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:53.007372 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
I20260810 06:22:53.204927 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:53.204504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:47789)
0810 06:22:53.204769 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:53.204822 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":99,"threads_started":1}
I20260810 06:22:53.896450 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:53.896017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:34743)
0810 06:22:53.896302 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:53.896355 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":78,"threads_started":1}
I20260810 06:22:54.008500 25573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:54.008137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:32791)
0810 06:22:54.008361 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:54.008413 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":71,"threads_started":1}
I20260810 06:22:54.205981 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:54.205553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:36891)
0810 06:22:54.205833 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:54.205894 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":73,"threads_started":1}
I20260810 06:22:54.596290 25576 raft_consensus.cc:482] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:22:54.596416 25576 raft_consensus.cc:504] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } } peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } }
I20260810 06:22:54.596772 25576 leader_election.cc:288] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bac5e0a95e9442bfa498f54d0b98c598 (127.31.204.195:38687), 807c1bfaa46c4ed098402fdf225ed4c6 (127.31.204.194:37759)
W20260810 06:22:54.597508 24931 leader_election.cc:341] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bac5e0a95e9442bfa498f54d0b98c598 (127.31.204.195:38687): Not found: Tablet not found: 3ad4f2aee14348ac9844cd86272080ae
W20260810 06:22:54.597630 24934 leader_election.cc:341] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 807c1bfaa46c4ed098402fdf225ed4c6 (127.31.204.194:37759): Not found: Tablet not found: 3ad4f2aee14348ac9844cd86272080ae
I20260810 06:22:54.597707 24934 leader_election.cc:302] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: befeb79e90b2436ea19b3a412dfe0c0a; no voters: 807c1bfaa46c4ed098402fdf225ed4c6, bac5e0a95e9442bfa498f54d0b98c598
I20260810 06:22:54.597891 25576 raft_consensus.cc:2726] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 06:22:54.897568 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:54.897149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:49703)
0810 06:22:54.897417 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:54.897470 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":97,"threads_started":1}
I20260810 06:22:55.009516 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:55.009145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:55405)
0810 06:22:55.009380 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:55.009429 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":81,"threads_started":1}
I20260810 06:22:55.207027 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:55.206634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:48615)
0810 06:22:55.206886 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:55.206940 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20260810 06:22:55.898729 25582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:55.898299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:53909)
0810 06:22:55.898554 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:55.898613 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260810 06:22:56.010555 25583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:56.010151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:54393)
0810 06:22:56.010414 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:56.010466 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":69,"threads_started":1}
I20260810 06:22:56.208271 25584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:56.207817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:47245)
0810 06:22:56.208105 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:56.208177 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":78,"threads_started":1}
I20260810 06:22:56.899835 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:56.899440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:56889)
0810 06:22:56.899683 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:56.899739 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260810 06:22:57.011505 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:57.011135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:33203)
0810 06:22:57.011372 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:57.011424 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":84,"threads_started":1}
I20260810 06:22:57.209323 25587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:57.208911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:34031)
0810 06:22:57.209183 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:57.209239 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":82,"threads_started":1}
I20260810 06:22:57.532738 25590 raft_consensus.cc:482] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:22:57.532874 25590 raft_consensus.cc:504] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } } peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } }
I20260810 06:22:57.533346 25590 leader_election.cc:288] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bac5e0a95e9442bfa498f54d0b98c598 (127.31.204.195:38687), 807c1bfaa46c4ed098402fdf225ed4c6 (127.31.204.194:37759)
W20260810 06:22:57.534058 24931 leader_election.cc:341] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bac5e0a95e9442bfa498f54d0b98c598 (127.31.204.195:38687): Not found: Tablet not found: 3ad4f2aee14348ac9844cd86272080ae
W20260810 06:22:57.534160 24934 leader_election.cc:341] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 807c1bfaa46c4ed098402fdf225ed4c6 (127.31.204.194:37759): Not found: Tablet not found: 3ad4f2aee14348ac9844cd86272080ae
I20260810 06:22:57.534248 24934 leader_election.cc:302] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: befeb79e90b2436ea19b3a412dfe0c0a; no voters: 807c1bfaa46c4ed098402fdf225ed4c6, bac5e0a95e9442bfa498f54d0b98c598
I20260810 06:22:57.534392 25590 raft_consensus.cc:2726] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 06:22:57.900856 25591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:57.900457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:46997)
0810 06:22:57.900707 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:57.900757 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
I20260810 06:22:58.012509 25592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:58.012111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:49557)
0810 06:22:58.012358 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:58.012415 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
I20260810 06:22:58.210398 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:58.209984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:60037)
0810 06:22:58.210256 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:58.210309 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":80,"threads_started":1}
I20260810 06:22:58.901991 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:58.901586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:59463)
0810 06:22:58.901837 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:58.901892 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260810 06:22:59.013628 25597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:59.013230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:33349)
0810 06:22:59.013475 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:59.013530 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":78,"threads_started":1}
I20260810 06:22:59.211525 25599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:59.211108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:45363)
0810 06:22:59.211378 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:59.211434 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":77,"threads_started":1}
I20260810 06:22:59.903043 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:22:59.902663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:38371)
0810 06:22:59.902894 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 06:22:59.902945 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
I20260810 06:23:00.014714 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:00.014309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:54513)
0810 06:23:00.014565 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:00.014624 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20260810 06:23:00.212527 25605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:00.212136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:43809)
0810 06:23:00.212387 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:00.212442 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":68,"threads_started":1}
I20260810 06:23:00.904186 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:00.903701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:40589)
0810 06:23:00.903988 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:00.904055 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":121,"threads_started":1}
I20260810 06:23:01.015688 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:01.015335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:60539)
0810 06:23:01.015543 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:01.015594 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
I20260810 06:23:01.213567 25615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:01.213175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:35441)
0810 06:23:01.213423 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:01.213476 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260810 06:23:01.905295 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:01.904887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:43701)
0810 06:23:01.905144 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:01.905199 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
I20260810 06:23:02.016806 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:02.016364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:49455)
0810 06:23:02.016643 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:02.016702 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":77,"threads_started":1}
I20260810 06:23:02.214622 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:02.214242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:38375)
0810 06:23:02.214479 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:02.214533 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20260810 06:23:02.906389 25623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:02.905929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:59075)
0810 06:23:02.906231 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:02.906289 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":100,"threads_started":1}
I20260810 06:23:03.017908 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:03.017533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:58321)
0810 06:23:03.017765 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:03.017819 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
I20260810 06:23:03.215632 25627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:03.215237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:38435)
0810 06:23:03.215490 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:03.215545 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260810 06:23:03.603747 25628 raft_consensus.cc:482] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:23:03.603870 25628 raft_consensus.cc:504] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5e0a95e9442bfa498f54d0b98c598" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 38687 } } peers { permanent_uuid: "807c1bfaa46c4ed098402fdf225ed4c6" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37759 } } peers { permanent_uuid: "befeb79e90b2436ea19b3a412dfe0c0a" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43547 } }
I20260810 06:23:03.604229 25628 leader_election.cc:288] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bac5e0a95e9442bfa498f54d0b98c598 (127.31.204.195:38687), 807c1bfaa46c4ed098402fdf225ed4c6 (127.31.204.194:37759)
W20260810 06:23:03.604910 24931 leader_election.cc:341] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bac5e0a95e9442bfa498f54d0b98c598 (127.31.204.195:38687): Not found: Tablet not found: 3ad4f2aee14348ac9844cd86272080ae
W20260810 06:23:03.605024 24934 leader_election.cc:341] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 807c1bfaa46c4ed098402fdf225ed4c6 (127.31.204.194:37759): Not found: Tablet not found: 3ad4f2aee14348ac9844cd86272080ae
I20260810 06:23:03.605093 24934 leader_election.cc:302] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: befeb79e90b2436ea19b3a412dfe0c0a; no voters: 807c1bfaa46c4ed098402fdf225ed4c6, bac5e0a95e9442bfa498f54d0b98c598
I20260810 06:23:03.605225 25628 raft_consensus.cc:2726] T 3ad4f2aee14348ac9844cd86272080ae P befeb79e90b2436ea19b3a412dfe0c0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 06:23:03.907599 25630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:03.907204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.193:46475)
0810 06:23:03.907454 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:03.907505 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":81,"threads_started":1}
I20260810 06:23:04.019047 25631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:04.018674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.194:46359)
0810 06:23:04.018900 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:04.018954 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":77,"threads_started":1}
I20260810 06:23:04.206413 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 24821
I20260810 06:23:04.216886 25632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:04.216490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:46155 (local address 127.31.204.195:36973)
0810 06:23:04.216734 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:04.216796 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":70,"threads_started":1}
I20260810 06:23:04.221155 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 25120
I20260810 06:23:04.232105 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 25335
2026-08-10T06:23:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19148 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:23:04 dist-test-slave-q9h9 krb5kdc[25635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:23:04 dist-test-slave-q9h9 krb5kdc[25635](info): set up 2 sockets
Aug 10 06:23:04 dist-test-slave-q9h9 krb5kdc[25635](info): commencing operation
krb5kdc: starting...
W20260810 06:23:05.393846 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:23:05 dist-test-slave-q9h9 krb5kdc[25635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:23:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:23:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:23:05.504117 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36891
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:36891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:23:05.612954 25658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:23:05.613215 25658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:23:05.613277 25658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:23:05.617223 25658 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 06:23:05.617298 25658 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 06:23:05.617331 25658 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 06:23:05.617352 25658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:23:05.617386 25658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:23:05.617426 25658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:23:05.617461 25658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:23:05.617496 25658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:23:05.622892 25658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:36891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:23:05.624078 25658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:23:05.625136 25658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:23:05.634963 25665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:23:05.635030 25658 server_base.cc:1034] running on GCE node
W20260810 06:23:05.635267 25663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:05.635253 25667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:23:05.635921 25658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:23:05.636968 25658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:23:05.638088 25658 hybrid_clock.cc:648] HybridClock initialized: now 1786342985638065 us; error 31 us; skew 500 ppm
Aug 10 06:23:05 dist-test-slave-q9h9 krb5kdc[25635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342985, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:23:05.640722 25658 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:23:05.641618 25658 webserver.cc:460] Webserver started at http://127.31.204.254:36827/ using document root <none> and password file <none>
I20260810 06:23:05.642184 25658 fs_manager.cc:359] Metadata directory not provided
I20260810 06:23:05.642257 25658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:23:05.642467 25658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:23:05.644464 25658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "1b05338d8dbe4a76979bc9a085bdeed9"
format_stamp: "Formatted at 2026-08-10 06:23:05 on dist-test-slave-q9h9"
server_key: "b48cfdd05f6226bc38e4ad25374430f1"
server_key_iv: "08ef8a3c24c03168f81c6e15e827180a"
server_key_version: "encryptionkey@0"
I20260810 06:23:05.645031 25658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "1b05338d8dbe4a76979bc9a085bdeed9"
format_stamp: "Formatted at 2026-08-10 06:23:05 on dist-test-slave-q9h9"
server_key: "b48cfdd05f6226bc38e4ad25374430f1"
server_key_iv: "08ef8a3c24c03168f81c6e15e827180a"
server_key_version: "encryptionkey@0"
I20260810 06:23:05.648758 25658 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 06:23:05.651324 25678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:05.652462 25658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:23:05.652583 25658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "1b05338d8dbe4a76979bc9a085bdeed9"
format_stamp: "Formatted at 2026-08-10 06:23:05 on dist-test-slave-q9h9"
server_key: "b48cfdd05f6226bc38e4ad25374430f1"
server_key_iv: "08ef8a3c24c03168f81c6e15e827180a"
server_key_version: "encryptionkey@0"
I20260810 06:23:05.652685 25658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:23:05.675976 25658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:23:05.678670 25658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:23:05.678828 25658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:23:05.685617 25658 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:36891
I20260810 06:23:05.685881 25758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:36891 every 8 connection(s)
I20260810 06:23:05.686569 25658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:23:05.689165 25760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:23:05.689217 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 25658
I20260810 06:23:05.689328 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:23:05.689646 32563 external_mini_cluster.cc:1428] Setting key 9ea6d7fa75480c9612ce870f1d6e1adb
I20260810 06:23:05.696075 25760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9: Bootstrap starting.
I20260810 06:23:05.698513 25760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9: Neither blocks nor log segments found. Creating new log.
Aug 10 06:23:05 dist-test-slave-q9h9 krb5kdc[25635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:23:05.699332 25760 log.cc:824] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9: Log is configured to *not* fsync() on all Append() calls
I20260810 06:23:05.701109 25760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9: No bootstrap required, opened a new log
I20260810 06:23:05.703648 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:05.690996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.31.204.254:36891)
0810 06:23:05.691404 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:05.691421 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:05.691461 (+ 40us) server_negotiation.cc:407] Connection header received
0810 06:23:05.692489 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:05.692513 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:05.692922 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:05.693278 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:23:05.693961 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:05.695420 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:05.696259 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:05.696906 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:05.697058 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:05.699637 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:23:05.699644 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:23:05.699659 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:23:05.699687 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:23:05.701129 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:05.701530 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:05.701535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:05.701539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:05.701641 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:05.701903 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:05.701906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:05.701908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:05.702307 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:23:05.702472 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:05.702740 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 06:23:05.703010 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":124,"threads_started":1}
I20260810 06:23:05.704639 25760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36891 } }
I20260810 06:23:05.704852 25760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:23:05.704924 25760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b05338d8dbe4a76979bc9a085bdeed9, State: Initialized, Role: FOLLOWER
I20260810 06:23:05.705338 25760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36891 } }
I20260810 06:23:05.705502 25760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:23:05.705615 25760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:23:05.705724 25760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:23:05.706617 25760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36891 } }
I20260810 06:23:05.706928 25760 leader_election.cc:302] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b05338d8dbe4a76979bc9a085bdeed9; no voters:
I20260810 06:23:05.707221 25760 leader_election.cc:288] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:23:05.707322 25767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:23:05.707522 25767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 1 LEADER]: Becoming Leader. State: Replica: 1b05338d8dbe4a76979bc9a085bdeed9, State: Running, Role: LEADER
I20260810 06:23:05.707844 25767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36891 } }
I20260810 06:23:05.708308 25760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:23:05.709288 25768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36891 } } }
I20260810 06:23:05.709421 25768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [sys.catalog]: This master's current role is: LEADER
I20260810 06:23:05.709766 25769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b05338d8dbe4a76979bc9a085bdeed9. Latest consensus state: current_term: 1 leader_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36891 } } }
I20260810 06:23:05.709857 25769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [sys.catalog]: This master's current role is: LEADER
I20260810 06:23:05.709947 25780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:23:05.712882 25780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:23:05.718313 25780 catalog_manager.cc:1347] Generated new cluster ID: 0005cdd6470944afaf922e0161d36d86
I20260810 06:23:05.718364 25780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:23:05.726711 25780 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:23:05.727489 25780 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:23:05.740767 25780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9: Generated new TSK 0
I20260810 06:23:05.741319 25780 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
I20260810 06:23:05.771724 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36891
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:23:05.877826 25800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:23:05.878065 25800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:23:05.878105 25800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:23:05.882216 25800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:23:05.882268 25800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:23:05.882333 25800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:23:05.887312 25800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:23:05.888214 25800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:23:05.889150 25800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:23:05.898126 25814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:05.898137 25811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:05.899048 25810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:23:05.899569 25800 server_base.cc:1034] running on GCE node
I20260810 06:23:05.899963 25800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:23:05.900545 25800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:23:05.901710 25800 hybrid_clock.cc:648] HybridClock initialized: now 1786342985901689 us; error 28 us; skew 500 ppm
Aug 10 06:23:05 dist-test-slave-q9h9 krb5kdc[25635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342985, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:23:05.904279 25800 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:23:05.905047 25800 webserver.cc:460] Webserver started at http://127.31.204.193:35943/ using document root <none> and password file <none>
I20260810 06:23:05.905534 25800 fs_manager.cc:359] Metadata directory not provided
I20260810 06:23:05.905594 25800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:23:05.905787 25800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:23:05.907759 25800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "bbd3c9c2e84b481fa51beb02c7c81753"
format_stamp: "Formatted at 2026-08-10 06:23:05 on dist-test-slave-q9h9"
server_key: "d3feac168f784dd3c35ce44c9fdff60e"
server_key_iv: "e8ebfa6dad46df50b61cf9898af58860"
server_key_version: "encryptionkey@0"
I20260810 06:23:05.908324 25800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "bbd3c9c2e84b481fa51beb02c7c81753"
format_stamp: "Formatted at 2026-08-10 06:23:05 on dist-test-slave-q9h9"
server_key: "d3feac168f784dd3c35ce44c9fdff60e"
server_key_iv: "e8ebfa6dad46df50b61cf9898af58860"
server_key_version: "encryptionkey@0"
I20260810 06:23:05.911860 25800 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:23:05.914003 25826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:05.915145 25800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:23:05.915241 25800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "bbd3c9c2e84b481fa51beb02c7c81753"
format_stamp: "Formatted at 2026-08-10 06:23:05 on dist-test-slave-q9h9"
server_key: "d3feac168f784dd3c35ce44c9fdff60e"
server_key_iv: "e8ebfa6dad46df50b61cf9898af58860"
server_key_version: "encryptionkey@0"
I20260810 06:23:05.915334 25800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:23:05.925401 25800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:23:05.927968 25800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:23:05.928125 25800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:23:05.928638 25800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:23:05.929382 25800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:23:05.929428 25800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:05.929469 25800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:23:05.929498 25800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:05.937551 25800 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:38105
I20260810 06:23:05.937574 26032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:38105 every 8 connection(s)
I20260810 06:23:05.938369 25800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
Aug 10 06:23:05 dist-test-slave-q9h9 krb5kdc[25635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342985, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:23:05.947028 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 25800
I20260810 06:23:05.947160 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:23:05.947504 32563 external_mini_cluster.cc:1428] Setting key f9d4863ca55267f9e976ce66b5f5dc24
I20260810 06:23:05.950899 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:05.939940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:36231 (local address 127.31.204.254:36891)
0810 06:23:05.940058 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:05.940062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:05.941001 (+ 939us) server_negotiation.cc:407] Connection header received
0810 06:23:05.941848 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:05.941851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:05.941908 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:05.941981 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:23:05.943085 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:05.943626 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:05.944727 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:05.944957 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:05.945044 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:05.947623 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:23:05.947627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:23:05.947629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:23:05.947653 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:23:05.948991 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:05.949524 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:05.949526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:05.949527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:05.949593 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:05.949918 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:05.949919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:05.949920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:05.950109 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:23:05.950183 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:05.950685 (+ 502us) server_negotiation.cc:299] Negotiation successful
0810 06:23:05.950786 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:23:05.951738 26035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:05.940282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:36231)
0810 06:23:05.940869 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:05.940907 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:05.941656 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:05.942176 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:23:05.942183 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:23:05.942510 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:23:05.942944 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:05.942958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:05.943751 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:05.943755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:05.944606 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:05.944615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:05.945094 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:05.945098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:05.945282 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:05.945802 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:23:05.945822 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:23:05.947457 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:23:05.949100 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:23:05.949106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:23:05.949119 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:23:05.949440 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:23:05.949659 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:23:05.949662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:23:05.949663 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:23:05.949842 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:23:05.950195 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:23:05.950201 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:23:05.950474 (+ 273us) client_negotiation.cc:769] Sending connection context
0810 06:23:05.950661 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 06:23:05.950970 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":98,"threads_started":1}
I20260810 06:23:05.952775 26033 heartbeater.cc:347] Connected to a master server at 127.31.204.254:36891
I20260810 06:23:05.952987 26033 heartbeater.cc:464] Registering TS with master...
I20260810 06:23:05.953421 26033 heartbeater.cc:511] Master 127.31.204.254:36891 requested a full tablet report, sending...
I20260810 06:23:05.954967 25706 ts_manager.cc:194] Registered new tserver with Master: bbd3c9c2e84b481fa51beb02c7c81753 (127.31.204.193:38105)
I20260810 06:23:05.956506 25706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:36231
WARNING: no policy specified for kudu/127.31.204.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
I20260810 06:23:05.973338 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36891
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:23:06.080505 26039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:23:06.080752 26039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:23:06.080798 26039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:23:06.084744 26039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:23:06.084797 26039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:23:06.084867 26039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:23:06.089917 26039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:23:06.090920 26039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:23:06.091897 26039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:23:06.101210 26050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:06.101243 26052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:06.101238 26049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:23:06.101434 26039 server_base.cc:1034] running on GCE node
I20260810 06:23:06.101907 26039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:23:06.102511 26039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:23:06.103662 26039 hybrid_clock.cc:648] HybridClock initialized: now 1786342986103645 us; error 28 us; skew 500 ppm
Aug 10 06:23:06 dist-test-slave-q9h9 krb5kdc[25635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342986, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:23:06.106276 26039 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:23:06.107038 26039 webserver.cc:460] Webserver started at http://127.31.204.194:36661/ using document root <none> and password file <none>
I20260810 06:23:06.107532 26039 fs_manager.cc:359] Metadata directory not provided
I20260810 06:23:06.107589 26039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:23:06.107748 26039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:23:06.109859 26039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "e7ef34e827a94f0785ee8b60691af394"
format_stamp: "Formatted at 2026-08-10 06:23:06 on dist-test-slave-q9h9"
server_key: "896fdc027f36579f2aa802952eaf5283"
server_key_iv: "7d4b439a9440cce912a6a370013e0e10"
server_key_version: "encryptionkey@0"
I20260810 06:23:06.110416 26039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "e7ef34e827a94f0785ee8b60691af394"
format_stamp: "Formatted at 2026-08-10 06:23:06 on dist-test-slave-q9h9"
server_key: "896fdc027f36579f2aa802952eaf5283"
server_key_iv: "7d4b439a9440cce912a6a370013e0e10"
server_key_version: "encryptionkey@0"
I20260810 06:23:06.114043 26039 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:23:06.116339 26063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:06.117492 26039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 06:23:06.117585 26039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "e7ef34e827a94f0785ee8b60691af394"
format_stamp: "Formatted at 2026-08-10 06:23:06 on dist-test-slave-q9h9"
server_key: "896fdc027f36579f2aa802952eaf5283"
server_key_iv: "7d4b439a9440cce912a6a370013e0e10"
server_key_version: "encryptionkey@0"
I20260810 06:23:06.117697 26039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:23:06.158674 26039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:23:06.161715 26039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:23:06.161891 26039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:23:06.162513 26039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:23:06.163326 26039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:23:06.163371 26039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:06.163416 26039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:23:06.163445 26039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:06.171496 26039 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:45605
I20260810 06:23:06.171535 26247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:45605 every 8 connection(s)
I20260810 06:23:06.172333 26039 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:23:06.179519 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 26039
I20260810 06:23:06.179631 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:23:06.179963 32563 external_mini_cluster.cc:1428] Setting key a345f628551c7db5008228bf048578a9
Aug 10 06:23:06 dist-test-slave-q9h9 krb5kdc[25635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342986, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:23:06.185360 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:06.174041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:44049 (local address 127.31.204.254:36891)
0810 06:23:06.174173 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:06.174177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:06.175176 (+ 999us) server_negotiation.cc:407] Connection header received
0810 06:23:06.175994 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:06.175996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:06.176042 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:06.176111 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:23:06.177294 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.177807 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.178869 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.179081 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.179152 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:06.181743 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:23:06.181746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:23:06.181748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:23:06.181774 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:23:06.183246 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:06.183774 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:06.183779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:06.183780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:06.183897 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:06.184250 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:06.184254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:06.184256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:06.184446 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:23:06.184542 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:06.185040 (+ 498us) server_negotiation.cc:299] Negotiation successful
0810 06:23:06.185184 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 06:23:06.186228 26250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:06.174395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:44049)
0810 06:23:06.175034 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:06.175075 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:06.175827 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:06.176335 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:23:06.176341 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:23:06.176682 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:23:06.177172 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:06.177189 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.177901 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.177903 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:06.178776 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:06.178785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.179243 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.179247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:06.179397 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:06.179961 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:23:06.179979 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:23:06.181557 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:23:06.183372 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:23:06.183379 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:23:06.183392 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:23:06.183668 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:23:06.183993 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:23:06.183997 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:23:06.183999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:23:06.184152 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:23:06.184563 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:23:06.184570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:23:06.184798 (+ 228us) client_negotiation.cc:769] Sending connection context
0810 06:23:06.184973 (+ 175us) client_negotiation.cc:240] Negotiation successful
0810 06:23:06.185280 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":115,"threads_started":1}
I20260810 06:23:06.187402 26248 heartbeater.cc:347] Connected to a master server at 127.31.204.254:36891
I20260810 06:23:06.187623 26248 heartbeater.cc:464] Registering TS with master...
I20260810 06:23:06.188105 26248 heartbeater.cc:511] Master 127.31.204.254:36891 requested a full tablet report, sending...
I20260810 06:23:06.189157 25706 ts_manager.cc:194] Registered new tserver with Master: e7ef34e827a94f0785ee8b60691af394 (127.31.204.194:45605)
WARNING: no policy specified for kudu/127.31.204.195@KRBTEST.COM; defaulting to no policy
I20260810 06:23:06.189913 25706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:44049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
I20260810 06:23:06.204535 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36891
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:23:06.315083 26254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:23:06.315325 26254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:23:06.315364 26254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:23:06.319262 26254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:23:06.319310 26254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:23:06.319381 26254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:23:06.324584 26254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:23:06.325556 26254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:23:06.326581 26254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:23:06.335769 26268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:06.335829 26266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:06.335825 26265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:23:06.336004 26254 server_base.cc:1034] running on GCE node
I20260810 06:23:06.336459 26254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:23:06.337076 26254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:23:06.338264 26254 hybrid_clock.cc:648] HybridClock initialized: now 1786342986338248 us; error 26 us; skew 500 ppm
Aug 10 06:23:06 dist-test-slave-q9h9 krb5kdc[25635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342986, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:23:06.340843 26254 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:23:06.341629 26254 webserver.cc:460] Webserver started at http://127.31.204.195:42169/ using document root <none> and password file <none>
I20260810 06:23:06.342185 26254 fs_manager.cc:359] Metadata directory not provided
I20260810 06:23:06.342247 26254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:23:06.342435 26254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:23:06.344380 26254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "33a74eeb639b4843aa79415f13aa5612"
format_stamp: "Formatted at 2026-08-10 06:23:06 on dist-test-slave-q9h9"
server_key: "71efb229ca356fe1e3b4c7c0d7b2fdfc"
server_key_iv: "22cbf960b4112ec32ae6a24cef510320"
server_key_version: "encryptionkey@0"
I20260810 06:23:06.344913 26254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "33a74eeb639b4843aa79415f13aa5612"
format_stamp: "Formatted at 2026-08-10 06:23:06 on dist-test-slave-q9h9"
server_key: "71efb229ca356fe1e3b4c7c0d7b2fdfc"
server_key_iv: "22cbf960b4112ec32ae6a24cef510320"
server_key_version: "encryptionkey@0"
I20260810 06:23:06.348398 26254 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260810 06:23:06.350617 26282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:06.351747 26254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:23:06.351861 26254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "33a74eeb639b4843aa79415f13aa5612"
format_stamp: "Formatted at 2026-08-10 06:23:06 on dist-test-slave-q9h9"
server_key: "71efb229ca356fe1e3b4c7c0d7b2fdfc"
server_key_iv: "22cbf960b4112ec32ae6a24cef510320"
server_key_version: "encryptionkey@0"
I20260810 06:23:06.351955 26254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:23:06.377671 26254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:23:06.380514 26254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:23:06.380678 26254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:23:06.381264 26254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:23:06.382092 26254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:23:06.382138 26254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:06.382176 26254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:23:06.382190 26254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:06.390980 26254 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:43099
I20260810 06:23:06.391010 26466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:43099 every 8 connection(s)
I20260810 06:23:06.391829 26254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
Aug 10 06:23:06 dist-test-slave-q9h9 krb5kdc[25635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786342986, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:23:06.400736 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 26254
I20260810 06:23:06.400857 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:23:06.401129 32563 external_mini_cluster.cc:1428] Setting key 5bc59803e01f45cbc99eedeafd98d7d6
I20260810 06:23:06.404245 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:06.393514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:36203 (local address 127.31.204.254:36891)
0810 06:23:06.393639 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:06.393642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:06.394623 (+ 981us) server_negotiation.cc:407] Connection header received
0810 06:23:06.395461 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:06.395464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:06.395512 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:06.395581 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:23:06.396661 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.397176 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.398244 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.398440 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.398507 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:06.400788 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:23:06.400791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:23:06.400794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:23:06.400826 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:23:06.402363 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:06.402914 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:06.402917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:06.402918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:06.403002 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:06.403362 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:06.403365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:06.403367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:06.403548 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:23:06.403624 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:06.404021 (+ 397us) server_negotiation.cc:299] Negotiation successful
0810 06:23:06.404114 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:23:06.404958 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:06.393829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:36203)
0810 06:23:06.394495 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:06.394535 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:06.395293 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:06.395725 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:23:06.395731 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:23:06.396075 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:23:06.396547 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:06.396561 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.397262 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.397264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:06.398150 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:06.398160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.398541 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.398544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:06.398675 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:06.399202 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:23:06.399218 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:23:06.400612 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:23:06.402481 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:23:06.402486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:23:06.402496 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:23:06.402831 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:23:06.403087 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:23:06.403090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:23:06.403092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:23:06.403267 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:23:06.403627 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:23:06.403632 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:23:06.403846 (+ 214us) client_negotiation.cc:769] Sending connection context
0810 06:23:06.403990 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 06:23:06.404212 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":143,"threads_started":1}
I20260810 06:23:06.405896 26467 heartbeater.cc:347] Connected to a master server at 127.31.204.254:36891
I20260810 06:23:06.406134 26467 heartbeater.cc:464] Registering TS with master...
I20260810 06:23:06.406580 26467 heartbeater.cc:511] Master 127.31.204.254:36891 requested a full tablet report, sending...
I20260810 06:23:06.407505 25706 ts_manager.cc:194] Registered new tserver with Master: 33a74eeb639b4843aa79415f13aa5612 (127.31.204.195:43099)
I20260810 06:23:06.408013 25706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:36203
I20260810 06:23:06.414749 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:23:06.423372 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:06.416571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.31.204.254:36891)
0810 06:23:06.416674 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:06.416678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:06.416829 (+ 151us) server_negotiation.cc:407] Connection header received
0810 06:23:06.416947 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:06.416949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:06.416984 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:06.417045 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:23:06.417635 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.418200 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.419011 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.419232 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.419308 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:06.420526 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:23:06.420530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:23:06.420533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:23:06.420558 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:23:06.421835 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:06.422288 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:06.422292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:06.422293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:06.422367 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:06.422606 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:06.422609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:06.422611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:06.422810 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:23:06.422922 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:06.423165 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 06:23:06.423265 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 06:23:06.425794 25706 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:23:06.428680 25706 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:23:06.441717 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:06.436413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:43099 (local address 127.0.0.1:40462)
0810 06:23:06.437205 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:06.437254 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:06.437379 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:06.437970 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:23:06.437976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:23:06.438000 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:23:06.438181 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:06.438188 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.439256 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.439266 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:06.440396 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:06.440406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.441147 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.441151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:06.441306 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:06.441349 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 06:23:06.441458 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 06:23:06.441570 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260810 06:23:06.441753 26492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:06.436549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40462 (local address 127.31.204.195:43099)
0810 06:23:06.437491 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:06.437503 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:06.437526 (+ 23us) server_negotiation.cc:407] Connection header received
0810 06:23:06.437600 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:06.437605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:06.437825 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:06.437982 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:23:06.438285 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.439140 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.440503 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.441039 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.441148 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:06.441283 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:06.441507 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 06:23:06.441607 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":120,"threads_started":1}
I20260810 06:23:06.442706 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:06.436665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:45605 (local address 127.0.0.1:59836)
0810 06:23:06.437580 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:06.437593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:06.437682 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:06.438392 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:23:06.438395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:23:06.438412 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:23:06.438513 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:06.438519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.440071 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.440073 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:06.441298 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:06.441306 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.442328 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.442331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:06.442414 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:06.442435 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 06:23:06.442512 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 06:23:06.442582 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":77,"thread_start_us":93,"threads_started":1}
I20260810 06:23:06.442770 26495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:06.436852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.31.204.194:45605)
0810 06:23:06.437298 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:06.437309 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:06.437678 (+ 369us) server_negotiation.cc:407] Connection header received
0810 06:23:06.437847 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:06.437853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:06.438072 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:06.438201 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:23:06.438638 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.439816 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.441402 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.442223 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.442338 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:06.442452 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:06.442538 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 06:23:06.442620 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":135,"threads_started":1}
I20260810 06:23:06.442853 26496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:06.437105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:38105 (local address 127.0.0.1:55522)
0810 06:23:06.437898 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:06.437910 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:06.437993 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:06.438506 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:23:06.438508 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:23:06.438528 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:23:06.438629 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:06.438635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.439979 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.439982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:06.441241 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:06.441253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.442452 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.442456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:06.442556 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:06.442580 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 06:23:06.442648 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 06:23:06.442730 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":4,"thread_start_us":81,"threads_started":1}
I20260810 06:23:06.443169 26497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:06.437508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.31.204.193:38105)
0810 06:23:06.437864 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:06.437881 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:06.438106 (+ 225us) server_negotiation.cc:407] Connection header received
0810 06:23:06.438191 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:06.438196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:06.438374 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:06.438518 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:23:06.438739 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.439836 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.441387 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.442149 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.442262 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:06.442385 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:06.442948 (+ 563us) server_negotiation.cc:299] Negotiation successful
0810 06:23:06.443038 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
I20260810 06:23:06.445358 26356 tablet_service.cc:1467] Processing CreateTablet for tablet 0e8db2f25bce459d9c08fcc7a5a1fa22 (DEFAULT_TABLE table=test-table [id=00196c8b4ccd42c3acb448fdeac8801f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:23:06.445791 26139 tablet_service.cc:1467] Processing CreateTablet for tablet 0e8db2f25bce459d9c08fcc7a5a1fa22 (DEFAULT_TABLE table=test-table [id=00196c8b4ccd42c3acb448fdeac8801f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:23:06.446550 26356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e8db2f25bce459d9c08fcc7a5a1fa22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:23:06.446691 26139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e8db2f25bce459d9c08fcc7a5a1fa22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:23:06.446779 25916 tablet_service.cc:1467] Processing CreateTablet for tablet 0e8db2f25bce459d9c08fcc7a5a1fa22 (DEFAULT_TABLE table=test-table [id=00196c8b4ccd42c3acb448fdeac8801f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:23:06.447717 25916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e8db2f25bce459d9c08fcc7a5a1fa22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:23:06.452687 26498 tablet_bootstrap.cc:492] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394: Bootstrap starting.
I20260810 06:23:06.454504 26500 tablet_bootstrap.cc:492] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753: Bootstrap starting.
I20260810 06:23:06.455341 26498 tablet_bootstrap.cc:654] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394: Neither blocks nor log segments found. Creating new log.
I20260810 06:23:06.455513 26499 tablet_bootstrap.cc:492] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612: Bootstrap starting.
I20260810 06:23:06.456228 26498 log.cc:824] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394: Log is configured to *not* fsync() on all Append() calls
I20260810 06:23:06.456655 26500 tablet_bootstrap.cc:654] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753: Neither blocks nor log segments found. Creating new log.
I20260810 06:23:06.457532 26500 log.cc:824] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753: Log is configured to *not* fsync() on all Append() calls
I20260810 06:23:06.457592 26499 tablet_bootstrap.cc:654] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612: Neither blocks nor log segments found. Creating new log.
I20260810 06:23:06.458174 26498 tablet_bootstrap.cc:492] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394: No bootstrap required, opened a new log
I20260810 06:23:06.458384 26498 ts_tablet_manager.cc:1397] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 06:23:06.458776 26499 log.cc:824] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612: Log is configured to *not* fsync() on all Append() calls
I20260810 06:23:06.459481 26500 tablet_bootstrap.cc:492] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753: No bootstrap required, opened a new log
I20260810 06:23:06.459673 26500 ts_tablet_manager.cc:1397] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:23:06.460763 26499 tablet_bootstrap.cc:492] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612: No bootstrap required, opened a new log
I20260810 06:23:06.461018 26499 ts_tablet_manager.cc:1397] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260810 06:23:06.461640 26498 raft_consensus.cc:348] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } }
I20260810 06:23:06.461866 26498 raft_consensus.cc:374] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:23:06.461937 26498 raft_consensus.cc:729] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ef34e827a94f0785ee8b60691af394, State: Initialized, Role: FOLLOWER
I20260810 06:23:06.462371 26498 consensus_queue.cc:260] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } }
I20260810 06:23:06.463014 26248 heartbeater.cc:503] Master 127.31.204.254:36891 was elected leader, sending a full tablet report...
I20260810 06:23:06.463275 26498 ts_tablet_manager.cc:1428] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 06:23:06.464318 26500 raft_consensus.cc:348] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } }
I20260810 06:23:06.464567 26500 raft_consensus.cc:374] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:23:06.464645 26500 raft_consensus.cc:729] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd3c9c2e84b481fa51beb02c7c81753, State: Initialized, Role: FOLLOWER
I20260810 06:23:06.464512 26499 raft_consensus.cc:348] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } }
I20260810 06:23:06.464771 26499 raft_consensus.cc:374] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:23:06.464828 26499 raft_consensus.cc:729] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a74eeb639b4843aa79415f13aa5612, State: Initialized, Role: FOLLOWER
I20260810 06:23:06.465076 26500 consensus_queue.cc:260] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } }
I20260810 06:23:06.465193 26499 consensus_queue.cc:260] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } }
I20260810 06:23:06.465746 26033 heartbeater.cc:503] Master 127.31.204.254:36891 was elected leader, sending a full tablet report...
I20260810 06:23:06.466080 26500 ts_tablet_manager.cc:1428] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 06:23:06.466150 26499 ts_tablet_manager.cc:1428] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 06:23:06.466502 26467 heartbeater.cc:503] Master 127.31.204.254:36891 was elected leader, sending a full tablet report...
W20260810 06:23:06.642730 26468 tablet.cc:2369] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:23:06.665285 26512 raft_consensus.cc:482] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:23:06.665478 26512 raft_consensus.cc:504] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } }
I20260810 06:23:06.666704 26512 leader_election.cc:288] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbd3c9c2e84b481fa51beb02c7c81753 (127.31.204.193:38105), e7ef34e827a94f0785ee8b60691af394 (127.31.204.194:45605)
I20260810 06:23:06.670785 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:06.666937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:38105 (local address 127.31.204.195:49631)
0810 06:23:06.667132 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:06.667146 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:06.667240 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:06.667476 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:23:06.667479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:23:06.667500 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:23:06.667635 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:06.667641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.668608 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.668628 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:06.669612 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:06.669623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.670399 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.670404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:06.670503 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:06.670528 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 06:23:06.670595 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 06:23:06.670658 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":46,"spinlock_wait_cycles":3712}
I20260810 06:23:06.670822 26497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:06.667097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:49631 (local address 127.31.204.193:38105)
0810 06:23:06.667229 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:06.667233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:06.667250 (+ 17us) server_negotiation.cc:407] Connection header received
0810 06:23:06.667331 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:06.667334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:06.667379 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:06.667444 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:23:06.667744 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.668510 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.669727 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.670294 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.670386 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:06.670427 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:06.670621 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 06:23:06.670699 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 06:23:06.670946 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:06.666973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:45605 (local address 127.31.204.195:36387)
0810 06:23:06.667420 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:06.667431 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:06.667532 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:06.667749 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:23:06.667751 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:23:06.667768 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:23:06.667850 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:06.667854 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.668753 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.668756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:06.669776 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:06.669787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.670549 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.670552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:06.670662 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:06.670685 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 06:23:06.670756 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 06:23:06.670817 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":1,"spinlock_wait_cycles":1280,"thread_start_us":151,"threads_started":1}
I20260810 06:23:06.670962 26495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:06.667102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:36387 (local address 127.31.204.194:45605)
0810 06:23:06.667219 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:06.667223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:06.667529 (+ 306us) server_negotiation.cc:407] Connection header received
0810 06:23:06.667620 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:06.667622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:06.667665 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:06.667738 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:23:06.667943 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.668648 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.669886 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.670429 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.670537 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:06.670583 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:06.670781 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 06:23:06.670856 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:23:06.671505 26173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e8db2f25bce459d9c08fcc7a5a1fa22" candidate_uuid: "33a74eeb639b4843aa79415f13aa5612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ef34e827a94f0785ee8b60691af394" is_pre_election: true
I20260810 06:23:06.671720 25950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e8db2f25bce459d9c08fcc7a5a1fa22" candidate_uuid: "33a74eeb639b4843aa79415f13aa5612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" is_pre_election: true
I20260810 06:23:06.671820 26173 raft_consensus.cc:2454] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33a74eeb639b4843aa79415f13aa5612 in term 0.
I20260810 06:23:06.672008 25950 raft_consensus.cc:2454] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33a74eeb639b4843aa79415f13aa5612 in term 0.
I20260810 06:23:06.672329 26289 leader_election.cc:302] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33a74eeb639b4843aa79415f13aa5612, e7ef34e827a94f0785ee8b60691af394; no voters:
I20260810 06:23:06.672550 26512 raft_consensus.cc:2781] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:23:06.672650 26512 raft_consensus.cc:482] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:23:06.672701 26512 raft_consensus.cc:3037] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [term 0 FOLLOWER]: Advancing to term 1
W20260810 06:23:06.673357 26249 tablet.cc:2369] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:23:06.673738 26512 raft_consensus.cc:504] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } }
I20260810 06:23:06.674070 26512 leader_election.cc:288] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [CANDIDATE]: Term 1 election: Requested vote from peers bbd3c9c2e84b481fa51beb02c7c81753 (127.31.204.193:38105), e7ef34e827a94f0785ee8b60691af394 (127.31.204.194:45605)
I20260810 06:23:06.674384 26173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e8db2f25bce459d9c08fcc7a5a1fa22" candidate_uuid: "33a74eeb639b4843aa79415f13aa5612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ef34e827a94f0785ee8b60691af394"
I20260810 06:23:06.674396 25950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e8db2f25bce459d9c08fcc7a5a1fa22" candidate_uuid: "33a74eeb639b4843aa79415f13aa5612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd3c9c2e84b481fa51beb02c7c81753"
I20260810 06:23:06.674489 26173 raft_consensus.cc:3037] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:23:06.674494 25950 raft_consensus.cc:3037] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:23:06.675457 26173 raft_consensus.cc:2454] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33a74eeb639b4843aa79415f13aa5612 in term 1.
I20260810 06:23:06.675462 25950 raft_consensus.cc:2454] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33a74eeb639b4843aa79415f13aa5612 in term 1.
I20260810 06:23:06.675732 26285 leader_election.cc:302] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33a74eeb639b4843aa79415f13aa5612, bbd3c9c2e84b481fa51beb02c7c81753; no voters:
I20260810 06:23:06.675933 26512 raft_consensus.cc:2781] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:23:06.676107 26512 raft_consensus.cc:686] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [term 1 LEADER]: Becoming Leader. State: Replica: 33a74eeb639b4843aa79415f13aa5612, State: Running, Role: LEADER
I20260810 06:23:06.676380 26512 consensus_queue.cc:237] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } }
I20260810 06:23:06.679299 25704 catalog_manager.cc:5495] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33a74eeb639b4843aa79415f13aa5612 (127.31.204.195). New cstate: current_term: 1 leader_uuid: "33a74eeb639b4843aa79415f13aa5612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } health_report { overall_health: UNKNOWN } } }
W20260810 06:23:06.690203 26034 tablet.cc:2369] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:23:06.733705 26492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:06.730375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40478 (local address 127.31.204.195:43099)
0810 06:23:06.730507 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:06.730512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:06.730525 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:23:06.730625 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:06.730627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:06.730679 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:06.730763 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:23:06.731006 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.731497 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.732333 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:06.732546 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:06.732653 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:06.732951 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:23:06.733332 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:23:06.733440 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:06.733512 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 06:23:06.733568 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 06:23:06.739112 26173 raft_consensus.cc:1261] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394 [term 1 FOLLOWER]: Refusing update from remote peer 33a74eeb639b4843aa79415f13aa5612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:23:06.739112 25950 raft_consensus.cc:1261] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Refusing update from remote peer 33a74eeb639b4843aa79415f13aa5612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:23:06.739701 26512 consensus_queue.cc:1035] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:23:06.739893 26515 consensus_queue.cc:1035] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:23:06.746867 26518 mvcc.cc:205] Tried to move back new op lower bound from 7316860873676644352 to 7316860873428606976. Current Snapshot: MvccSnapshot[applied={T|T < 7316860873676644352}]
I20260810 06:23:06.748095 26523 mvcc.cc:205] Tried to move back new op lower bound from 7316860873676644352 to 7316860873428606976. Current Snapshot: MvccSnapshot[applied={T|T < 7316860873676644352}]
I20260810 06:23:06.748687 26528 mvcc.cc:205] Tried to move back new op lower bound from 7316860873676644352 to 7316860873428606976. Current Snapshot: MvccSnapshot[applied={T|T < 7316860873676644352}]
I20260810 06:23:06.755620 25704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:23:06.755774 25704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:23:06.758100 25704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9: Sending DeleteTablet for 3 replicas of tablet 0e8db2f25bce459d9c08fcc7a5a1fa22
I20260810 06:23:06.758833 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 25800
I20260810 06:23:06.758838 25916 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e8db2f25bce459d9c08fcc7a5a1fa22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:23:06 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:55522
I20260810 06:23:06.758868 26139 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e8db2f25bce459d9c08fcc7a5a1fa22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:23:06 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:59836
I20260810 06:23:06.766417 26356 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e8db2f25bce459d9c08fcc7a5a1fa22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:23:06 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:40462
I20260810 06:23:06.767356 26540 tablet_replica.cc:330] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612: stopping tablet replica
I20260810 06:23:06.767426 26539 tablet_replica.cc:330] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394: stopping tablet replica
I20260810 06:23:06.767697 26540 raft_consensus.cc:2229] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:23:06.767741 26539 raft_consensus.cc:2229] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:23:06.768067 26540 raft_consensus.cc:2258] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:23:06.768077 26539 raft_consensus.cc:2258] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:23:06.769414 26539 ts_tablet_manager.cc:1905] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:23:06.770462 26540 ts_tablet_manager.cc:1905] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:23:06.772140 26539 ts_tablet_manager.cc:1918] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:23:06.772274 26539 log.cc:1196] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/0e8db2f25bce459d9c08fcc7a5a1fa22
I20260810 06:23:06.772624 26539 ts_tablet_manager.cc:1939] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P e7ef34e827a94f0785ee8b60691af394: Deleting consensus metadata
I20260810 06:23:06.773126 26540 ts_tablet_manager.cc:1918] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:23:06.773255 26540 log.cc:1196] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/0e8db2f25bce459d9c08fcc7a5a1fa22
I20260810 06:23:06.773588 26540 ts_tablet_manager.cc:1939] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P 33a74eeb639b4843aa79415f13aa5612: Deleting consensus metadata
I20260810 06:23:06.773897 25683 catalog_manager.cc:4841] TS e7ef34e827a94f0785ee8b60691af394 (127.31.204.194:45605): tablet 0e8db2f25bce459d9c08fcc7a5a1fa22 (table test-table [id=00196c8b4ccd42c3acb448fdeac8801f]) successfully deleted
I20260810 06:23:06.774636 25679 catalog_manager.cc:4841] TS 33a74eeb639b4843aa79415f13aa5612 (127.31.204.195:43099): tablet 0e8db2f25bce459d9c08fcc7a5a1fa22 (table test-table [id=00196c8b4ccd42c3acb448fdeac8801f]) successfully deleted
W20260810 06:23:06.775887 25679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.204.193:38105 (error 108)
I20260810 06:23:06.776675 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 26039
I20260810 06:23:06.776785 26496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:06.776528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:38105 (local address 127.0.0.1:55538)
0810 06:23:06.776649 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:06.776705 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:38105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 06:23:06.810643 25679 catalog_manager.cc:4584] TS bbd3c9c2e84b481fa51beb02c7c81753 (127.31.204.193:38105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0e8db2f25bce459d9c08fcc7a5a1fa22: Network error: Client connection negotiation failed: client connection to 127.31.204.193:38105: connect: Connection refused (error 111)
I20260810 06:23:06.820328 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 26254
I20260810 06:23:06.833989 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 25658
I20260810 06:23:26.845345 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36891
--webserver_interface=127.31.204.254
--webserver_port=36827
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:36891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:23:26.953970 26542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:23:26.954206 26542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:23:26.954243 26542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:23:26.958055 26542 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 06:23:26.958091 26542 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 06:23:26.958105 26542 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 06:23:26.958117 26542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:23:26.958133 26542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:23:26.958154 26542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:23:26.958173 26542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:23:26.958199 26542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:23:26.963142 26542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:36891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=36827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:23:26.963999 26542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:23:26.964883 26542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:23:26.977870 26552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:26.977870 26556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:23:26.977968 26542 server_base.cc:1034] running on GCE node
W20260810 06:23:26.977972 26553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:23:26.978747 26542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:23:26.979776 26542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:23:26.980929 26542 hybrid_clock.cc:648] HybridClock initialized: now 1786343006980913 us; error 27 us; skew 500 ppm
Aug 10 06:23:26 dist-test-slave-q9h9 krb5kdc[25635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343006, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:23:26.983556 26542 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:23:26.984393 26542 webserver.cc:460] Webserver started at http://127.31.204.254:36827/ using document root <none> and password file <none>
I20260810 06:23:26.984920 26542 fs_manager.cc:359] Metadata directory not provided
I20260810 06:23:26.985000 26542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:23:26.988607 26542 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:23:26.990686 26564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:26.991819 26542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:23:26.991966 26542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "1b05338d8dbe4a76979bc9a085bdeed9"
format_stamp: "Formatted at 2026-08-10 06:23:05 on dist-test-slave-q9h9"
server_key: "b48cfdd05f6226bc38e4ad25374430f1"
server_key_iv: "08ef8a3c24c03168f81c6e15e827180a"
server_key_version: "encryptionkey@0"
I20260810 06:23:26.992520 26542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:23:27.013168 26542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:23:27.015869 26542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:23:27.016034 26542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:23:27.022632 26542 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:36891
I20260810 06:23:27.022893 26642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:36891 every 8 connection(s)
I20260810 06:23:27.023631 26542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:23:27.027901 26645 sys_catalog.cc:263] Verifying existing consensus state
I20260810 06:23:27.029464 26645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9: Bootstrap starting.
I20260810 06:23:27.032595 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 26542
I20260810 06:23:27.033000 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:38105
--local_ip_for_outbound_sockets=127.31.204.193
--tserver_master_addrs=127.31.204.254:36891
--webserver_port=35943
--webserver_interface=127.31.204.193
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:23:27.050343 26645 log.cc:824] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9: Log is configured to *not* fsync() on all Append() calls
I20260810 06:23:27.060783 26645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:23:27.061187 26645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9: Bootstrap complete.
I20260810 06:23:27.066525 26645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36891 } }
I20260810 06:23:27.067022 26645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b05338d8dbe4a76979bc9a085bdeed9, State: Initialized, Role: FOLLOWER
I20260810 06:23:27.067554 26645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36891 } }
I20260810 06:23:27.067706 26645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:23:27.067778 26645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:23:27.067873 26645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 1 FOLLOWER]: Advancing to term 2
I20260810 06:23:27.070230 26645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36891 } }
I20260810 06:23:27.070577 26645 leader_election.cc:302] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b05338d8dbe4a76979bc9a085bdeed9; no voters:
I20260810 06:23:27.070925 26645 leader_election.cc:288] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 06:23:27.071027 26654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 06:23:27.071272 26654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 2 LEADER]: Becoming Leader. State: Replica: 1b05338d8dbe4a76979bc9a085bdeed9, State: Running, Role: LEADER
I20260810 06:23:27.071609 26654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36891 } }
I20260810 06:23:27.072108 26645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:23:27.072932 26659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b05338d8dbe4a76979bc9a085bdeed9. Latest consensus state: current_term: 2 leader_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36891 } } }
I20260810 06:23:27.072969 26656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36891 } } }
I20260810 06:23:27.073057 26656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [sys.catalog]: This master's current role is: LEADER
I20260810 06:23:27.073057 26659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [sys.catalog]: This master's current role is: LEADER
I20260810 06:23:27.073592 26669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:23:27.078155 26669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:23:27.078899 26669 catalog_manager.cc:1259] Loaded cluster ID: 0005cdd6470944afaf922e0161d36d86
I20260810 06:23:27.078962 26669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:23:27.081395 26669 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:23:27.082113 26669 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9: Loaded TSK: 0
I20260810 06:23:27.093729 26669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9: Generated new TSK 1
I20260810 06:23:27.093967 26669 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 06:23:27.156245 26650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:23:27.156490 26650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:23:27.156550 26650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:23:27.160495 26650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:23:27.160552 26650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:23:27.160629 26650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:23:27.165714 26650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:38105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=35943
--tserver_master_addrs=127.31.204.254:36891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:23:27.166676 26650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:23:27.167651 26650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:23:27.178421 26691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:27.178464 26695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:27.178781 26693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:23:27.179086 26650 server_base.cc:1034] running on GCE node
I20260810 06:23:27.179566 26650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:23:27.180356 26650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:23:27.181568 26650 hybrid_clock.cc:648] HybridClock initialized: now 1786343007181530 us; error 41 us; skew 500 ppm
Aug 10 06:23:27 dist-test-slave-q9h9 krb5kdc[25635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343007, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:23:27.184636 26650 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:23:27.185439 26650 webserver.cc:460] Webserver started at http://127.31.204.193:35943/ using document root <none> and password file <none>
I20260810 06:23:27.185992 26650 fs_manager.cc:359] Metadata directory not provided
I20260810 06:23:27.186115 26650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:23:27.189855 26650 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:23:27.192062 26703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:27.193324 26650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:23:27.193430 26650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "bbd3c9c2e84b481fa51beb02c7c81753"
format_stamp: "Formatted at 2026-08-10 06:23:05 on dist-test-slave-q9h9"
server_key: "d3feac168f784dd3c35ce44c9fdff60e"
server_key_iv: "e8ebfa6dad46df50b61cf9898af58860"
server_key_version: "encryptionkey@0"
I20260810 06:23:27.193945 26650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:23:27.213917 26650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:23:27.216735 26650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:23:27.216917 26650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:23:27.217486 26650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:23:27.218555 26716 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 06:23:27.222551 26650 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 06:23:27.222604 26650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260810 06:23:27.222662 26650 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 06:23:27.224236 26650 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 06:23:27.224280 26650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 06:23:27.224552 26716 tablet_bootstrap.cc:492] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753: Bootstrap starting.
I20260810 06:23:27.233414 26650 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:38105
I20260810 06:23:27.233479 26879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:38105 every 8 connection(s)
I20260810 06:23:27.234371 26650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:23:27.240896 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 26650
I20260810 06:23:27.241312 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:45605
--local_ip_for_outbound_sockets=127.31.204.194
--tserver_master_addrs=127.31.204.254:36891
--webserver_port=36661
--webserver_interface=127.31.204.194
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:23:27.243386 26716 log.cc:824] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753: Log is configured to *not* fsync() on all Append() calls
Aug 10 06:23:27 dist-test-slave-q9h9 krb5kdc[25635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343007, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:23:27.251330 26716 tablet_bootstrap.cc:492] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:23:27.251895 26716 tablet_bootstrap.cc:492] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753: Bootstrap complete.
I20260810 06:23:27.252434 26883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:27.236323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:50075)
0810 06:23:27.236803 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:27.236856 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:27.237657 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:27.239027 (+ 1370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:23:27.239033 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:23:27.239378 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:23:27.239819 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:27.239832 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.241718 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.241721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:27.242959 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:27.242971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.243942 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.243946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:27.244143 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:27.244793 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:23:27.244812 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:23:27.246684 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:23:27.249095 (+ 2411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:23:27.249101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:23:27.249113 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:23:27.249443 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:23:27.249839 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:23:27.249842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:23:27.249844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:23:27.250048 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:23:27.250722 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:23:27.250729 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:23:27.250997 (+ 268us) client_negotiation.cc:769] Sending connection context
0810 06:23:27.251185 (+ 188us) client_negotiation.cc:240] Negotiation successful
0810 06:23:27.251503 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":126,"threads_started":1}
I20260810 06:23:27.252496 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:27.236380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:50075 (local address 127.31.204.254:36891)
0810 06:23:27.236912 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:27.236930 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:27.236982 (+ 52us) server_negotiation.cc:407] Connection header received
0810 06:23:27.238105 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:27.238122 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:27.238596 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:27.239004 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:23:27.240000 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.241555 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.243095 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.243779 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.244516 (+ 737us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:27.246903 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:23:27.246911 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:23:27.246928 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:23:27.246960 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:23:27.248967 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:27.249561 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:27.249567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:27.249573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:27.249727 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:27.250133 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:27.250136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:27.250138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:27.250590 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:23:27.250731 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:27.251346 (+ 615us) server_negotiation.cc:299] Negotiation successful
0810 06:23:27.251649 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":142,"threads_started":1}
I20260810 06:23:27.253247 26716 ts_tablet_manager.cc:1397] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753: Time spent bootstrapping tablet: real 0.029s user 0.027s sys 0.000s
I20260810 06:23:27.255607 26880 heartbeater.cc:347] Connected to a master server at 127.31.204.254:36891
I20260810 06:23:27.255849 26880 heartbeater.cc:464] Registering TS with master...
I20260810 06:23:27.255824 26716 raft_consensus.cc:348] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } }
I20260810 06:23:27.256258 26716 raft_consensus.cc:729] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd3c9c2e84b481fa51beb02c7c81753, State: Initialized, Role: FOLLOWER
I20260810 06:23:27.256361 26880 heartbeater.cc:511] Master 127.31.204.254:36891 requested a full tablet report, sending...
I20260810 06:23:27.256745 26716 consensus_queue.cc:260] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } }
I20260810 06:23:27.257694 26716 ts_tablet_manager.cc:1428] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260810 06:23:27.259143 26592 ts_manager.cc:194] Registered new tserver with Master: bbd3c9c2e84b481fa51beb02c7c81753 (127.31.204.193:38105)
I20260810 06:23:27.262365 26592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:50075
I20260810 06:23:27.264966 26880 heartbeater.cc:503] Master 127.31.204.254:36891 was elected leader, sending a full tablet report...
Aug 10 06:23:27 dist-test-slave-q9h9 krb5kdc[25635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343006, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for kudu/127.31.204.193@KRBTEST.COM
I20260810 06:23:27.271634 26889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:27.261920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:38105 (local address 127.0.0.1:55664)
0810 06:23:27.262472 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:27.262514 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:27.262632 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:27.263267 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:23:27.263273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:23:27.263624 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:23:27.263728 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:27.263735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.264658 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.264661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:27.265360 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:27.265371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.266083 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.266087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:27.266202 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:27.266799 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:23:27.266813 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:23:27.268361 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:23:27.269960 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:23:27.269969 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:23:27.269974 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:23:27.270243 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:23:27.270516 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:23:27.270519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:23:27.270521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:23:27.270601 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:23:27.271024 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:23:27.271030 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:23:27.271141 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 06:23:27.271255 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 06:23:27.271443 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":160,"threads_started":1}
I20260810 06:23:27.271660 26891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:27.261979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55664 (local address 127.31.204.193:38105)
0810 06:23:27.262686 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:27.262696 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:27.262718 (+ 22us) server_negotiation.cc:407] Connection header received
0810 06:23:27.262790 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:27.262795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:27.262973 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:27.263093 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:23:27.263850 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.264551 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.265451 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.265916 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.266044 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:27.268494 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:23:27.268503 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:23:27.268512 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:23:27.268533 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:23:27.269858 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:27.270346 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:27.270351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:27.270352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:27.270431 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:27.270696 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:27.270699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:27.270700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:27.270913 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:23:27.271030 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:27.271369 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 06:23:27.271524 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":113,"threads_started":1}
I20260810 06:23:27.272802 26773 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e8db2f25bce459d9c08fcc7a5a1fa22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:23:06 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:55664
I20260810 06:23:27.273751 26892 tablet_replica.cc:330] stopping tablet replica
I20260810 06:23:27.274008 26892 raft_consensus.cc:2229] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:23:27.274250 26892 raft_consensus.cc:2258] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:23:27.275153 26892 ts_tablet_manager.cc:1905] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:23:27.278554 26892 ts_tablet_manager.cc:1918] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:23:27.278642 26892 log.cc:1196] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/wal/wals/0e8db2f25bce459d9c08fcc7a5a1fa22
I20260810 06:23:27.278887 26892 ts_tablet_manager.cc:1939] T 0e8db2f25bce459d9c08fcc7a5a1fa22 P bbd3c9c2e84b481fa51beb02c7c81753: Deleting consensus metadata
I20260810 06:23:27.280364 26565 catalog_manager.cc:4841] TS bbd3c9c2e84b481fa51beb02c7c81753 (127.31.204.193:38105): tablet 0e8db2f25bce459d9c08fcc7a5a1fa22 (table test-table [id=00196c8b4ccd42c3acb448fdeac8801f]) successfully deleted
W20260810 06:23:27.356554 26886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:23:27.356777 26886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:23:27.356817 26886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:23:27.360780 26886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:23:27.360832 26886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:23:27.360896 26886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:23:27.365890 26886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:45605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=36661
--tserver_master_addrs=127.31.204.254:36891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:23:27.366837 26886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:23:27.367787 26886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:23:27.377012 26901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:27.377055 26905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:27.377012 26900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:23:27.377291 26886 server_base.cc:1034] running on GCE node
I20260810 06:23:27.377765 26886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:23:27.378414 26886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:23:27.379562 26886 hybrid_clock.cc:648] HybridClock initialized: now 1786343007379536 us; error 45 us; skew 500 ppm
Aug 10 06:23:27 dist-test-slave-q9h9 krb5kdc[25635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343007, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:23:27.382103 26886 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:23:27.382891 26886 webserver.cc:460] Webserver started at http://127.31.204.194:36661/ using document root <none> and password file <none>
I20260810 06:23:27.383399 26886 fs_manager.cc:359] Metadata directory not provided
I20260810 06:23:27.383481 26886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:23:27.387068 26886 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260810 06:23:27.388938 26913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:27.390071 26886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:23:27.390170 26886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "e7ef34e827a94f0785ee8b60691af394"
format_stamp: "Formatted at 2026-08-10 06:23:06 on dist-test-slave-q9h9"
server_key: "896fdc027f36579f2aa802952eaf5283"
server_key_iv: "7d4b439a9440cce912a6a370013e0e10"
server_key_version: "encryptionkey@0"
I20260810 06:23:27.390661 26886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:23:27.402275 26886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:23:27.405020 26886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:23:27.405179 26886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:23:27.405702 26886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:23:27.406483 26886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:23:27.406530 26886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:27.406579 26886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:23:27.406605 26886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:27.414942 26886 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:45605
I20260810 06:23:27.414974 27099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:45605 every 8 connection(s)
I20260810 06:23:27.415882 26886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:23:27.418785 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 26886
I20260810 06:23:27.419193 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:43099
--local_ip_for_outbound_sockets=127.31.204.195
--tserver_master_addrs=127.31.204.254:36891
--webserver_port=42169
--webserver_interface=127.31.204.195
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 06:23:27 dist-test-slave-q9h9 krb5kdc[25635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343007, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:23:27.429159 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:27.417565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:47535 (local address 127.31.204.254:36891)
0810 06:23:27.417718 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:27.417721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:27.418722 (+ 1001us) server_negotiation.cc:407] Connection header received
0810 06:23:27.419634 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:27.419638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:27.419710 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:27.419783 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:23:27.420992 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.421521 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.422808 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.423032 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.423107 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:27.425701 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:23:27.425706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:23:27.425709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:23:27.425740 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:23:27.427231 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:27.427693 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:27.427697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:27.427699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:27.427793 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:27.428074 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:27.428077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:27.428078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:27.428312 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:23:27.428421 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:27.428842 (+ 421us) server_negotiation.cc:299] Negotiation successful
0810 06:23:27.428987 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 06:23:27.429912 27103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:27.417893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:47535)
0810 06:23:27.418574 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:27.418613 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:27.419439 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:27.419963 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:23:27.419970 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:23:27.420324 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:23:27.420808 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:27.420828 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.421641 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.421644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:27.422715 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:27.422724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.423178 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.423182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:27.423370 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:27.423932 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:23:27.423952 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:23:27.425521 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:23:27.427342 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:23:27.427348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:23:27.427358 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:23:27.427597 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:23:27.427877 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:23:27.427879 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:23:27.427880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:23:27.427994 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:23:27.428404 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:23:27.428408 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:23:27.428627 (+ 219us) client_negotiation.cc:769] Sending connection context
0810 06:23:27.428777 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 06:23:27.429009 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":150,"threads_started":1}
I20260810 06:23:27.431059 27101 heartbeater.cc:347] Connected to a master server at 127.31.204.254:36891
I20260810 06:23:27.431272 27101 heartbeater.cc:464] Registering TS with master...
I20260810 06:23:27.431712 27101 heartbeater.cc:511] Master 127.31.204.254:36891 requested a full tablet report, sending...
I20260810 06:23:27.432701 26592 ts_manager.cc:194] Registered new tserver with Master: e7ef34e827a94f0785ee8b60691af394 (127.31.204.194:45605)
I20260810 06:23:27.433439 26592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:47535
W20260810 06:23:27.531627 27109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:23:27.531869 27109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:23:27.531909 27109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:23:27.535835 27109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:23:27.535887 27109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:23:27.535940 27109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:23:27.540954 27109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:43099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=42169
--tserver_master_addrs=127.31.204.254:36891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:23:27.541898 27109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:23:27.542905 27109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:23:27.551955 27124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:23:27.552120 27109 server_base.cc:1034] running on GCE node
W20260810 06:23:27.551949 27120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:27.551982 27116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:23:27.552541 27109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:23:27.553125 27109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:23:27.554272 27109 hybrid_clock.cc:648] HybridClock initialized: now 1786343007554251 us; error 38 us; skew 500 ppm
Aug 10 06:23:27 dist-test-slave-q9h9 krb5kdc[25635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343007, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:23:27.556911 27109 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:23:27.557685 27109 webserver.cc:460] Webserver started at http://127.31.204.195:42169/ using document root <none> and password file <none>
I20260810 06:23:27.558245 27109 fs_manager.cc:359] Metadata directory not provided
I20260810 06:23:27.558317 27109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:23:27.562570 27109 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:23:27.564478 27137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:27.565625 27109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:23:27.565716 27109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "33a74eeb639b4843aa79415f13aa5612"
format_stamp: "Formatted at 2026-08-10 06:23:06 on dist-test-slave-q9h9"
server_key: "71efb229ca356fe1e3b4c7c0d7b2fdfc"
server_key_iv: "22cbf960b4112ec32ae6a24cef510320"
server_key_version: "encryptionkey@0"
I20260810 06:23:27.566270 27109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:23:27.584591 27109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:23:27.587766 27109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:23:27.587936 27109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:23:27.588498 27109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:23:27.589331 27109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:23:27.589382 27109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:27.589425 27109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:23:27.589449 27109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:27.599349 27109 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:43099
I20260810 06:23:27.599395 27309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:43099 every 8 connection(s)
I20260810 06:23:27.600347 27109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:23:27.607388 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 27109
Aug 10 06:23:27 dist-test-slave-q9h9 krb5kdc[25635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343007, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:23:27.613265 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:27.602137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:47421 (local address 127.31.204.254:36891)
0810 06:23:27.602277 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:27.602281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:27.603280 (+ 999us) server_negotiation.cc:407] Connection header received
0810 06:23:27.604079 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:27.604081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:27.604127 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:27.604186 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:23:27.605290 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.605788 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.606903 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.607111 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.607185 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:27.609692 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:23:27.609696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:23:27.609698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:23:27.609719 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:23:27.611192 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:27.611730 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:27.611733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:27.611734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:27.611814 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:27.612167 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:27.612169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:27.612170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:27.612330 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:23:27.612397 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:27.612941 (+ 544us) server_negotiation.cc:299] Negotiation successful
0810 06:23:27.613103 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 06:23:27.613955 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:27.602502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:47421)
0810 06:23:27.603112 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:27.603151 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:27.603897 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:27.604356 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:23:27.604362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:23:27.604706 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:23:27.605162 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:27.605174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.605888 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.605891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:27.606810 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:27.606820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.607266 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.607271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:27.607490 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:27.608059 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:23:27.608080 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:23:27.609522 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:23:27.611289 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:23:27.611295 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:23:27.611309 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:23:27.611635 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:23:27.611901 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:23:27.611905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:23:27.611906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:23:27.612080 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:23:27.612421 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:23:27.612428 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:23:27.612692 (+ 264us) client_negotiation.cc:769] Sending connection context
0810 06:23:27.612858 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 06:23:27.613158 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":105,"threads_started":1}
I20260810 06:23:27.615074 27310 heartbeater.cc:347] Connected to a master server at 127.31.204.254:36891
I20260810 06:23:27.615334 27310 heartbeater.cc:464] Registering TS with master...
I20260810 06:23:27.615793 27310 heartbeater.cc:511] Master 127.31.204.254:36891 requested a full tablet report, sending...
I20260810 06:23:27.616008 27315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:27.608055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53130 (local address 127.31.204.254:36891)
0810 06:23:27.608300 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:27.608304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:27.608444 (+ 140us) server_negotiation.cc:407] Connection header received
0810 06:23:27.608558 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:27.608562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:27.608601 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:27.608674 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:23:27.609426 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.609938 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.610817 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.611158 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.611266 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:27.612573 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:23:27.612576 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:23:27.612578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:23:27.612596 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:23:27.614083 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:27.614852 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:27.614855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:27.614857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:27.614935 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:27.615215 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:27.615218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:27.615220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:27.615400 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:23:27.615599 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:27.615735 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 06:23:27.615853 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":84,"threads_started":1}
I20260810 06:23:27.616806 26592 ts_manager.cc:194] Registered new tserver with Master: 33a74eeb639b4843aa79415f13aa5612 (127.31.204.195:43099)
I20260810 06:23:27.617326 26592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:47421
I20260810 06:23:27.617668 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:23:27.621579 27315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:27.618271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53146 (local address 127.31.204.254:36891)
0810 06:23:27.618379 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:27.618383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:27.618587 (+ 204us) server_negotiation.cc:407] Connection header received
0810 06:23:27.618719 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:27.618722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:27.618817 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:27.618900 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:23:27.619130 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.619613 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.620478 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.620706 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.620805 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:27.621096 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:23:27.621218 (+ 122us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 06:23:27.621462 (+ 244us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53146: token expired
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 06:23:27.621650 27315 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53146: token expired
W20260810 06:23:27.622282 26482 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.31.204.254:36891)
I20260810 06:23:27.630471 27315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:27.622680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53162 (local address 127.31.204.254:36891)
0810 06:23:27.622782 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:27.622784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:27.622796 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:23:27.622891 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:27.622893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:27.622925 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:27.622988 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:23:27.623699 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.624188 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.625017 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.625252 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.625336 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:27.627007 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:23:27.627011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:23:27.627014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:23:27.627040 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:23:27.628476 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:27.628947 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:27.628950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:27.628952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:27.629036 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:27.629317 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:27.629319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:27.629321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:27.629520 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:23:27.629646 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:27.630173 (+ 527us) server_negotiation.cc:299] Negotiation successful
0810 06:23:27.630321 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 06:23:27.633570 26592 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:23:27.637010 26592 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:23:27.644791 26773 tablet_service.cc:1467] Processing CreateTablet for tablet f9164f5d6e694a61b68c088f773a4bd1 (DEFAULT_TABLE table=test-table [id=3c954f662e30429683584c38ecae9ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:23:27.645476 26773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9164f5d6e694a61b68c088f773a4bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:23:27.647967 26889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:27.643313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:43099 (local address 127.0.0.1:55760)
0810 06:23:27.643433 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:27.643448 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:27.643523 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:27.644496 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:23:27.644499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:23:27.644518 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:23:27.644611 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:27.644617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.645644 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.645647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:27.646707 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:27.646717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.647547 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.647550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:27.647642 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:27.647684 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 06:23:27.647767 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 06:23:27.647842 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":4}
I20260810 06:23:27.648010 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:27.643647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.31.204.195:43099)
0810 06:23:27.644026 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:27.644039 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:27.644061 (+ 22us) server_negotiation.cc:407] Connection header received
0810 06:23:27.644130 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:27.644135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:27.644349 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:27.644506 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:23:27.644708 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.645531 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.646839 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.647431 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.647548 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:27.647683 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:27.647796 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 06:23:27.647885 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":169,"threads_started":1}
I20260810 06:23:27.648398 26716 tablet_bootstrap.cc:492] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753: Bootstrap starting.
I20260810 06:23:27.648988 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:27.643594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:45605 (local address 127.0.0.1:46182)
0810 06:23:27.643840 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:27.643855 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:27.643942 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:27.644832 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:23:27.644835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:23:27.644853 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:23:27.644933 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:27.644939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.646201 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.646204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:27.647428 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:27.647440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.648639 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.648641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:27.648706 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:27.648732 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 06:23:27.648794 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 06:23:27.648860 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":150,"thread_start_us":114,"threads_started":1}
I20260810 06:23:27.649065 27329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:27.644036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.31.204.194:45605)
0810 06:23:27.644383 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:27.644394 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:27.644417 (+ 23us) server_negotiation.cc:407] Connection header received
0810 06:23:27.644497 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:27.644502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:27.644686 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:27.644857 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:23:27.645035 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.646070 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.647577 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.648377 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.648491 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:27.648584 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:27.648848 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 06:23:27.648940 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":162,"threads_started":1}
I20260810 06:23:27.649289 26716 tablet_bootstrap.cc:654] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753: Neither blocks nor log segments found. Creating new log.
I20260810 06:23:27.651160 26716 tablet_bootstrap.cc:492] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753: No bootstrap required, opened a new log
I20260810 06:23:27.651257 26716 ts_tablet_manager.cc:1397] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 06:23:27.651286 27207 tablet_service.cc:1467] Processing CreateTablet for tablet f9164f5d6e694a61b68c088f773a4bd1 (DEFAULT_TABLE table=test-table [id=3c954f662e30429683584c38ecae9ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:23:27.651705 26716 raft_consensus.cc:348] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } }
I20260810 06:23:27.651815 26716 raft_consensus.cc:374] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:23:27.651862 26716 raft_consensus.cc:729] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd3c9c2e84b481fa51beb02c7c81753, State: Initialized, Role: FOLLOWER
I20260810 06:23:27.651990 26716 consensus_queue.cc:260] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } }
I20260810 06:23:27.651741 26995 tablet_service.cc:1467] Processing CreateTablet for tablet f9164f5d6e694a61b68c088f773a4bd1 (DEFAULT_TABLE table=test-table [id=3c954f662e30429683584c38ecae9ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:23:27.652248 26716 ts_tablet_manager.cc:1428] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:23:27.652428 27207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9164f5d6e694a61b68c088f773a4bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:23:27.652710 26995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9164f5d6e694a61b68c088f773a4bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:23:27.659403 27333 tablet_bootstrap.cc:492] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612: Bootstrap starting.
I20260810 06:23:27.660068 27335 tablet_bootstrap.cc:492] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394: Bootstrap starting.
I20260810 06:23:27.662398 27335 tablet_bootstrap.cc:654] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394: Neither blocks nor log segments found. Creating new log.
I20260810 06:23:27.663287 27335 log.cc:824] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394: Log is configured to *not* fsync() on all Append() calls
I20260810 06:23:27.664482 27333 tablet_bootstrap.cc:654] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612: Neither blocks nor log segments found. Creating new log.
I20260810 06:23:27.665305 27333 log.cc:824] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612: Log is configured to *not* fsync() on all Append() calls
I20260810 06:23:27.665369 27335 tablet_bootstrap.cc:492] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394: No bootstrap required, opened a new log
I20260810 06:23:27.665541 27335 ts_tablet_manager.cc:1397] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:23:27.667244 27333 tablet_bootstrap.cc:492] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612: No bootstrap required, opened a new log
I20260810 06:23:27.667434 27333 ts_tablet_manager.cc:1397] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260810 06:23:27.669266 27335 raft_consensus.cc:348] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } }
I20260810 06:23:27.669584 27335 raft_consensus.cc:374] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:23:27.669648 27335 raft_consensus.cc:729] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ef34e827a94f0785ee8b60691af394, State: Initialized, Role: FOLLOWER
I20260810 06:23:27.670133 27335 consensus_queue.cc:260] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } }
I20260810 06:23:27.670744 27101 heartbeater.cc:503] Master 127.31.204.254:36891 was elected leader, sending a full tablet report...
I20260810 06:23:27.671173 27335 ts_tablet_manager.cc:1428] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 06:23:27.671051 27333 raft_consensus.cc:348] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } }
I20260810 06:23:27.671321 27333 raft_consensus.cc:374] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:23:27.671378 27333 raft_consensus.cc:729] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a74eeb639b4843aa79415f13aa5612, State: Initialized, Role: FOLLOWER
I20260810 06:23:27.671764 27333 consensus_queue.cc:260] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } }
I20260810 06:23:27.672381 27310 heartbeater.cc:503] Master 127.31.204.254:36891 was elected leader, sending a full tablet report...
I20260810 06:23:27.672715 27333 ts_tablet_manager.cc:1428] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260810 06:23:27.735540 26882 tablet.cc:2369] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:23:27.851282 27311 tablet.cc:2369] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:23:27.916899 27102 tablet.cc:2369] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:23:27.980935 27338 raft_consensus.cc:482] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:23:27.981108 27338 raft_consensus.cc:504] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } }
I20260810 06:23:27.982254 27338 leader_election.cc:288] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33a74eeb639b4843aa79415f13aa5612 (127.31.204.195:43099), bbd3c9c2e84b481fa51beb02c7c81753 (127.31.204.193:38105)
I20260810 06:23:27.986069 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:27.982543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:40519 (local address 127.31.204.195:43099)
0810 06:23:27.982676 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:27.982680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:27.982872 (+ 192us) server_negotiation.cc:407] Connection header received
0810 06:23:27.982928 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:27.982931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:27.982978 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:27.983050 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:23:27.983358 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.983963 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.985048 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.985507 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.985607 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:27.985646 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:27.985859 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 06:23:27.985920 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:23:27.986088 27344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:27.982432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:43099 (local address 127.31.204.194:40519)
0810 06:23:27.982688 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:27.982702 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:27.982814 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:27.983103 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:23:27.983106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:23:27.983129 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:23:27.983261 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:27.983268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.984062 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.984066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:27.984935 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:27.984944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.985617 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.985621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:27.985734 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:27.985763 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 06:23:27.985839 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 06:23:27.985909 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":105,"threads_started":1}
I20260810 06:23:27.986364 27346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:27.982793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:38105 (local address 127.31.204.194:35559)
0810 06:23:27.982996 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:27.983008 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:27.983081 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:27.983435 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:23:27.983437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:23:27.983455 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:23:27.983582 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:27.983589 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.984361 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.984363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:27.985172 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:27.985181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.985901 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.985905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:27.986003 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:27.986059 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 06:23:27.986124 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 06:23:27.986225 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":63,"threads_started":1}
I20260810 06:23:27.986516 27347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:27.982936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:35559 (local address 127.31.204.193:38105)
0810 06:23:27.983179 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:27.983183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:27.983196 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:23:27.983257 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:27.983260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:27.983321 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:27.983416 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:23:27.983687 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.984263 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.985298 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:27.985794 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:27.985897 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:27.986285 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:27.986325 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 06:23:27.986394 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260810 06:23:27.986831 27236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9164f5d6e694a61b68c088f773a4bd1" candidate_uuid: "e7ef34e827a94f0785ee8b60691af394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a74eeb639b4843aa79415f13aa5612" is_pre_election: true
I20260810 06:23:27.986934 26804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9164f5d6e694a61b68c088f773a4bd1" candidate_uuid: "e7ef34e827a94f0785ee8b60691af394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" is_pre_election: true
I20260810 06:23:27.987123 27236 raft_consensus.cc:2454] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ef34e827a94f0785ee8b60691af394 in term 0.
I20260810 06:23:27.987169 26804 raft_consensus.cc:2454] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ef34e827a94f0785ee8b60691af394 in term 0.
I20260810 06:23:27.987584 26914 leader_election.cc:302] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33a74eeb639b4843aa79415f13aa5612, e7ef34e827a94f0785ee8b60691af394; no voters:
I20260810 06:23:27.987802 27338 raft_consensus.cc:2781] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:23:27.987879 27338 raft_consensus.cc:482] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:23:27.987928 27338 raft_consensus.cc:3037] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:23:27.988919 27338 raft_consensus.cc:504] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } }
I20260810 06:23:27.989205 27338 leader_election.cc:288] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [CANDIDATE]: Term 1 election: Requested vote from peers 33a74eeb639b4843aa79415f13aa5612 (127.31.204.195:43099), bbd3c9c2e84b481fa51beb02c7c81753 (127.31.204.193:38105)
I20260810 06:23:27.989547 27236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9164f5d6e694a61b68c088f773a4bd1" candidate_uuid: "e7ef34e827a94f0785ee8b60691af394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a74eeb639b4843aa79415f13aa5612"
I20260810 06:23:27.989568 26804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9164f5d6e694a61b68c088f773a4bd1" candidate_uuid: "e7ef34e827a94f0785ee8b60691af394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd3c9c2e84b481fa51beb02c7c81753"
I20260810 06:23:27.989655 27236 raft_consensus.cc:3037] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:23:27.989671 26804 raft_consensus.cc:3037] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:23:27.990672 26804 raft_consensus.cc:2454] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ef34e827a94f0785ee8b60691af394 in term 1.
I20260810 06:23:27.990680 27236 raft_consensus.cc:2454] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ef34e827a94f0785ee8b60691af394 in term 1.
I20260810 06:23:27.990926 26914 leader_election.cc:302] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbd3c9c2e84b481fa51beb02c7c81753, e7ef34e827a94f0785ee8b60691af394; no voters:
I20260810 06:23:27.991070 27338 raft_consensus.cc:2781] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:23:27.991274 27338 raft_consensus.cc:686] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [term 1 LEADER]: Becoming Leader. State: Replica: e7ef34e827a94f0785ee8b60691af394, State: Running, Role: LEADER
I20260810 06:23:27.991554 27338 consensus_queue.cc:237] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } }
I20260810 06:23:27.994567 26592 catalog_manager.cc:5495] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7ef34e827a94f0785ee8b60691af394 (127.31.204.194). New cstate: current_term: 1 leader_uuid: "e7ef34e827a94f0785ee8b60691af394" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } health_report { overall_health: UNKNOWN } } }
I20260810 06:23:28.007360 27329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:28.004619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39932 (local address 127.31.204.194:45605)
0810 06:23:28.004713 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:28.004717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:28.004730 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:23:28.004842 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:28.004844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:28.004888 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:28.004964 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:23:28.005108 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:28.005557 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:28.006300 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:28.006492 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:28.006559 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:28.006763 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:23:28.007101 (+ 338us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:23:28.007182 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:28.007207 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 06:23:28.007255 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 06:23:28.012414 27236 raft_consensus.cc:1261] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612 [term 1 FOLLOWER]: Refusing update from remote peer e7ef34e827a94f0785ee8b60691af394: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:23:28.012431 26804 raft_consensus.cc:1261] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Refusing update from remote peer e7ef34e827a94f0785ee8b60691af394: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:23:28.012993 27353 consensus_queue.cc:1035] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:23:28.013142 27338 consensus_queue.cc:1035] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:23:28.018857 27358 mvcc.cc:205] Tried to move back new op lower bound from 7316860960813043712 to 7316860960735789056. Current Snapshot: MvccSnapshot[applied={T|T < 7316860960813043712}]
I20260810 06:23:28.020176 27361 mvcc.cc:205] Tried to move back new op lower bound from 7316860960813043712 to 7316860960735789056. Current Snapshot: MvccSnapshot[applied={T|T < 7316860960813043712 or (T in {7316860960813043712})}]
I20260810 06:23:28.027951 26592 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:23:28.028091 26592 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:23:28.030355 26592 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9: Sending DeleteTablet for 3 replicas of tablet f9164f5d6e694a61b68c088f773a4bd1
I20260810 06:23:28.031039 27207 tablet_service.cc:1514] Processing DeleteTablet for tablet f9164f5d6e694a61b68c088f773a4bd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:23:28 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:55760
I20260810 06:23:28.031210 26995 tablet_service.cc:1514] Processing DeleteTablet for tablet f9164f5d6e694a61b68c088f773a4bd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:23:28 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:46182
I20260810 06:23:28.031440 26773 tablet_service.cc:1514] Processing DeleteTablet for tablet f9164f5d6e694a61b68c088f773a4bd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:23:28 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:55664
I20260810 06:23:28.031705 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 26650
I20260810 06:23:28.058897 27376 tablet_replica.cc:330] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394: stopping tablet replica
I20260810 06:23:28.059266 27376 raft_consensus.cc:2229] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:23:28.059640 27376 raft_consensus.cc:2258] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:23:28.062215 27376 ts_tablet_manager.cc:1905] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:23:28.064845 27376 ts_tablet_manager.cc:1918] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:23:28.064954 27376 log.cc:1196] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/f9164f5d6e694a61b68c088f773a4bd1
I20260810 06:23:28.065243 27376 ts_tablet_manager.cc:1939] T f9164f5d6e694a61b68c088f773a4bd1 P e7ef34e827a94f0785ee8b60691af394: Deleting consensus metadata
I20260810 06:23:28.066169 27374 tablet_replica.cc:330] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612: stopping tablet replica
I20260810 06:23:28.066474 26571 catalog_manager.cc:4841] TS e7ef34e827a94f0785ee8b60691af394 (127.31.204.194:45605): tablet f9164f5d6e694a61b68c088f773a4bd1 (table test-table [id=3c954f662e30429683584c38ecae9ab3]) successfully deleted
I20260810 06:23:28.066542 27374 raft_consensus.cc:2229] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:23:28.066844 27374 raft_consensus.cc:2258] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:23:28.068378 27374 ts_tablet_manager.cc:1905] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:23:28.071002 27374 ts_tablet_manager.cc:1918] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:23:28.071098 27374 log.cc:1196] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/f9164f5d6e694a61b68c088f773a4bd1
I20260810 06:23:28.071457 27374 ts_tablet_manager.cc:1939] T f9164f5d6e694a61b68c088f773a4bd1 P 33a74eeb639b4843aa79415f13aa5612: Deleting consensus metadata
W20260810 06:23:28.071960 26565 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.204.193:38105 (error 108)
I20260810 06:23:28.073055 26565 catalog_manager.cc:4841] TS 33a74eeb639b4843aa79415f13aa5612 (127.31.204.195:43099): tablet f9164f5d6e694a61b68c088f773a4bd1 (table test-table [id=3c954f662e30429683584c38ecae9ab3]) successfully deleted
W20260810 06:23:28.073438 27327 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:23:28.072922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:38105 (local address 127.0.0.1:32872)
0810 06:23:28.073050 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:28.073064 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:28.073173 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:28.073350 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:38105: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 06:23:28.073593 26565 catalog_manager.cc:4584] TS bbd3c9c2e84b481fa51beb02c7c81753 (127.31.204.193:38105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f9164f5d6e694a61b68c088f773a4bd1: Network error: Client connection negotiation failed: client connection to 127.31.204.193:38105: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 06:23:28.073629 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 26886
I20260810 06:23:28.084806 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 27109
I20260810 06:23:28.094076 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 26542
I20260810 06:23:48.108713 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36891
--webserver_interface=127.31.204.254
--webserver_port=36827
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:36891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:23:48.224502 27379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:23:48.224740 27379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:23:48.224794 27379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:23:48.228708 27379 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 06:23:48.228754 27379 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 06:23:48.228776 27379 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 06:23:48.228792 27379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:23:48.228816 27379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:23:48.228850 27379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:23:48.228874 27379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:23:48.228902 27379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:23:48.233870 27379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:36891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=36827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:23:48.234834 27379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:23:48.235776 27379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:23:48.243599 27387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:48.243641 27388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:48.243610 27390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:23:48.243777 27379 server_base.cc:1034] running on GCE node
I20260810 06:23:48.244412 27379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:23:48.245397 27379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:23:48.246563 27379 hybrid_clock.cc:648] HybridClock initialized: now 1786343028246544 us; error 37 us; skew 500 ppm
Aug 10 06:23:48 dist-test-slave-q9h9 krb5kdc[25635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343028, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:23:48.254374 27379 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:23:48.255206 27379 webserver.cc:460] Webserver started at http://127.31.204.254:36827/ using document root <none> and password file <none>
I20260810 06:23:48.255712 27379 fs_manager.cc:359] Metadata directory not provided
I20260810 06:23:48.255777 27379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:23:48.259335 27379 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:23:48.261267 27402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:48.262422 27379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:23:48.262528 27379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "1b05338d8dbe4a76979bc9a085bdeed9"
format_stamp: "Formatted at 2026-08-10 06:23:05 on dist-test-slave-q9h9"
server_key: "b48cfdd05f6226bc38e4ad25374430f1"
server_key_iv: "08ef8a3c24c03168f81c6e15e827180a"
server_key_version: "encryptionkey@0"
I20260810 06:23:48.263017 27379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:23:48.284991 27379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:23:48.287505 27379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:23:48.287663 27379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:23:48.294390 27379 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:36891
I20260810 06:23:48.294734 27500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:36891 every 8 connection(s)
I20260810 06:23:48.295456 27379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:23:48.296828 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 27379
I20260810 06:23:48.297242 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:38105
--local_ip_for_outbound_sockets=127.31.204.193
--tserver_master_addrs=127.31.204.254:36891
--webserver_port=35943
--webserver_interface=127.31.204.193
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:23:48.299773 27502 sys_catalog.cc:263] Verifying existing consensus state
I20260810 06:23:48.301469 27502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9: Bootstrap starting.
I20260810 06:23:48.319545 27502 log.cc:824] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9: Log is configured to *not* fsync() on all Append() calls
I20260810 06:23:48.330444 27502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:23:48.330789 27502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9: Bootstrap complete.
I20260810 06:23:48.334630 27502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36891 } }
I20260810 06:23:48.335022 27502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b05338d8dbe4a76979bc9a085bdeed9, State: Initialized, Role: FOLLOWER
I20260810 06:23:48.335458 27502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36891 } }
I20260810 06:23:48.335578 27502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:23:48.335644 27502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:23:48.335742 27502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 2 FOLLOWER]: Advancing to term 3
I20260810 06:23:48.337484 27502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36891 } }
I20260810 06:23:48.337790 27502 leader_election.cc:302] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b05338d8dbe4a76979bc9a085bdeed9; no voters:
I20260810 06:23:48.338088 27502 leader_election.cc:288] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 06:23:48.338178 27510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 06:23:48.338395 27510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [term 3 LEADER]: Becoming Leader. State: Replica: 1b05338d8dbe4a76979bc9a085bdeed9, State: Running, Role: LEADER
I20260810 06:23:48.338743 27510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36891 } }
I20260810 06:23:48.339286 27502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:23:48.340332 27511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36891 } } }
I20260810 06:23:48.340473 27511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [sys.catalog]: This master's current role is: LEADER
I20260810 06:23:48.340646 27513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b05338d8dbe4a76979bc9a085bdeed9. Latest consensus state: current_term: 3 leader_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b05338d8dbe4a76979bc9a085bdeed9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36891 } } }
I20260810 06:23:48.340749 27513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9 [sys.catalog]: This master's current role is: LEADER
I20260810 06:23:48.341236 27530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:23:48.346117 27530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:23:48.346797 27530 catalog_manager.cc:1259] Loaded cluster ID: 0005cdd6470944afaf922e0161d36d86
I20260810 06:23:48.346858 27530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:23:48.349290 27530 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:23:48.349996 27530 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9: Loaded TSK: 0
I20260810 06:23:48.350157 27530 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1b05338d8dbe4a76979bc9a085bdeed9: Loaded TSK: 1
F20260810 06:23:48.351341 27530 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe238cdd9ad google::LogMessage::Fail() at ??:0
@ 0x7fe238ce088a google::LogMessage::SendToLog() at ??:0
@ 0x7fe238cdd4de google::LogMessage::Flush() at ??:0
@ 0x7fe238cdea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe23b9a118d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe23b999f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe2436467bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe24362099c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe243621178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fe243662df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe24368182b std::function<>::operator()() at ??:0
@ 0x7fe243620b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe243621f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe24361e579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe243662640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe23af2f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe23af2fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe23af313f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe23af20512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe23c5646db start_thread at ??:0
@ 0x7fe2377db61f clone at ??:0
W20260810 06:23:48.424752 27505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:23:48.424984 27505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:23:48.425040 27505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:23:48.429222 27505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:23:48.429279 27505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:23:48.429342 27505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:23:48.434701 27505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:38105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=35943
--tserver_master_addrs=127.31.204.254:36891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:23:48.435647 27505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:23:48.436625 27505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:23:48.446182 27577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:48.446182 27576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:48.446558 27580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:23:48.447026 27505 server_base.cc:1034] running on GCE node
I20260810 06:23:48.447366 27505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:23:48.447925 27505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:23:48.449090 27505 hybrid_clock.cc:648] HybridClock initialized: now 1786343028449067 us; error 32 us; skew 500 ppm
Aug 10 06:23:48 dist-test-slave-q9h9 krb5kdc[25635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343028, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:23:48.451673 27505 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:23:48.452545 27505 webserver.cc:460] Webserver started at http://127.31.204.193:35943/ using document root <none> and password file <none>
I20260810 06:23:48.453039 27505 fs_manager.cc:359] Metadata directory not provided
I20260810 06:23:48.453099 27505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:23:48.457053 27505 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:23:48.459319 27590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:48.460556 27505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:23:48.460664 27505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "bbd3c9c2e84b481fa51beb02c7c81753"
format_stamp: "Formatted at 2026-08-10 06:23:05 on dist-test-slave-q9h9"
server_key: "d3feac168f784dd3c35ce44c9fdff60e"
server_key_iv: "e8ebfa6dad46df50b61cf9898af58860"
server_key_version: "encryptionkey@0"
I20260810 06:23:48.461190 27505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:23:48.476518 27505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:23:48.479488 27505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:23:48.479668 27505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:23:48.480242 27505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:23:48.481359 27598 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 06:23:48.486719 27505 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 06:23:48.486776 27505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260810 06:23:48.486848 27505 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 06:23:48.488474 27505 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 06:23:48.488517 27505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 06:23:48.488835 27598 tablet_bootstrap.cc:492] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753: Bootstrap starting.
I20260810 06:23:48.498517 27505 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:38105
I20260810 06:23:48.498644 27771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:38105 every 8 connection(s)
I20260810 06:23:48.499454 27505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:23:48.504406 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 27505
I20260810 06:23:48.504833 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:45605
--local_ip_for_outbound_sockets=127.31.204.194
--tserver_master_addrs=127.31.204.254:36891
--webserver_port=36661
--webserver_interface=127.31.204.194
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:23:48.510751 27598 log.cc:824] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753: Log is configured to *not* fsync() on all Append() calls
I20260810 06:23:48.519861 27598 tablet_bootstrap.cc:492] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:23:48.520298 27598 tablet_bootstrap.cc:492] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753: Bootstrap complete.
I20260810 06:23:48.521661 27598 ts_tablet_manager.cc:1397] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.004s
I20260810 06:23:48.524806 27598 raft_consensus.cc:348] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } }
I20260810 06:23:48.525342 27598 raft_consensus.cc:729] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd3c9c2e84b481fa51beb02c7c81753, State: Initialized, Role: FOLLOWER
I20260810 06:23:48.525851 27598 consensus_queue.cc:260] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } }
I20260810 06:23:48.526881 27598 ts_tablet_manager.cc:1428] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260810 06:23:48.582242 27777 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:23:48.501325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:35569)
0810 06:23:48.501809 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:48.501861 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:48.502685 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:48.581143 (+ 78458us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
W20260810 06:23:48.582768 27773 heartbeater.cc:644] Failed to heartbeat to 127.31.204.254:36891 (0 consecutive failures): Network error: Failed to ping master at 127.31.204.254:36891: Client connection negotiation failed: client connection to 127.31.204.254:36891: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 06:23:48.583531 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:48.583231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:53507)
0810 06:23:48.583377 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:48.583460 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 06:23:48.584213 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:48.583987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:36649)
0810 06:23:48.584091 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:48.584141 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 06:23:48.584383 27773 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 06:23:48.615487 27778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:23:48.615710 27778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:23:48.615751 27778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:23:48.619678 27778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:23:48.619730 27778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:23:48.619784 27778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:23:48.624965 27778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:45605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=36661
--tserver_master_addrs=127.31.204.254:36891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:23:48.626289 27778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:23:48.627381 27778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:23:48.636777 27793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:48.636740 27792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:48.636756 27796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:23:48.636996 27778 server_base.cc:1034] running on GCE node
I20260810 06:23:48.637439 27778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:23:48.638047 27778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:23:48.639225 27778 hybrid_clock.cc:648] HybridClock initialized: now 1786343028639203 us; error 33 us; skew 500 ppm
Aug 10 06:23:48 dist-test-slave-q9h9 krb5kdc[25635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343028, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:23:48.641765 27778 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:23:48.642722 27778 webserver.cc:460] Webserver started at http://127.31.204.194:36661/ using document root <none> and password file <none>
I20260810 06:23:48.643246 27778 fs_manager.cc:359] Metadata directory not provided
I20260810 06:23:48.643307 27778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:23:48.646854 27778 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:23:48.648736 27808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:48.649828 27778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:23:48.649923 27778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "e7ef34e827a94f0785ee8b60691af394"
format_stamp: "Formatted at 2026-08-10 06:23:06 on dist-test-slave-q9h9"
server_key: "896fdc027f36579f2aa802952eaf5283"
server_key_iv: "7d4b439a9440cce912a6a370013e0e10"
server_key_version: "encryptionkey@0"
I20260810 06:23:48.650465 27778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:23:48.665704 27778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:23:48.668553 27778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:23:48.668717 27778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:23:48.669250 27778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:23:48.670003 27778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:23:48.670073 27778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:48.670114 27778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:23:48.670145 27778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:48.678735 27778 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:45605
I20260810 06:23:48.678772 28008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:45605 every 8 connection(s)
I20260810 06:23:48.679679 27778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:23:48.681514 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 27778
I20260810 06:23:48.681933 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:43099
--local_ip_for_outbound_sockets=127.31.204.195
--tserver_master_addrs=127.31.204.254:36891
--webserver_port=42169
--webserver_interface=127.31.204.195
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:23:48.683470 28014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:48.681679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:36735)
0810 06:23:48.682434 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:48.682568 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":107,"threads_started":1}
W20260810 06:23:48.683902 28009 heartbeater.cc:644] Failed to heartbeat to 127.31.204.254:36891 (0 consecutive failures): Network error: Failed to ping master at 127.31.204.254:36891: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
I20260810 06:23:48.684549 28014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:48.684304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:55219)
0810 06:23:48.684427 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:48.684485 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 06:23:48.685169 28014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:48.684950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:53749)
0810 06:23:48.685054 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:48.685104 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 06:23:48.685438 28009 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 06:23:48.790618 28015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:23:48.790848 28015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:23:48.790901 28015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:23:48.794787 28015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:23:48.794853 28015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:23:48.794927 28015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:23:48.799899 28015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:43099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=42169
--tserver_master_addrs=127.31.204.254:36891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:23:48.800818 28015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:23:48.801802 28015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:23:48.810948 28020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:48.810967 28026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:23:48.811132 28023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:23:48.811206 28015 server_base.cc:1034] running on GCE node
I20260810 06:23:48.811582 28015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:23:48.812126 28015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:23:48.813274 28015 hybrid_clock.cc:648] HybridClock initialized: now 1786343028813256 us; error 29 us; skew 500 ppm
Aug 10 06:23:48 dist-test-slave-q9h9 krb5kdc[25635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343028, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:23:48.815874 28015 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:23:48.816672 28015 webserver.cc:460] Webserver started at http://127.31.204.195:42169/ using document root <none> and password file <none>
I20260810 06:23:48.817173 28015 fs_manager.cc:359] Metadata directory not provided
I20260810 06:23:48.817232 28015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:23:48.820833 28015 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:23:48.822755 28040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:48.823920 28015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:23:48.824024 28015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "33a74eeb639b4843aa79415f13aa5612"
format_stamp: "Formatted at 2026-08-10 06:23:06 on dist-test-slave-q9h9"
server_key: "71efb229ca356fe1e3b4c7c0d7b2fdfc"
server_key_iv: "22cbf960b4112ec32ae6a24cef510320"
server_key_version: "encryptionkey@0"
I20260810 06:23:48.824525 28015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:23:48.837888 28015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:23:48.840334 28015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:23:48.840493 28015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:23:48.841022 28015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:23:48.841765 28015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:23:48.841810 28015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:48.841861 28015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:23:48.841889 28015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:23:48.850344 28015 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:43099
I20260810 06:23:48.850368 28229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:43099 every 8 connection(s)
I20260810 06:23:48.851246 28015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:23:48.854576 28235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:48.853153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:60457)
0810 06:23:48.853765 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:48.853862 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":111,"threads_started":1}
W20260810 06:23:48.855023 28230 heartbeater.cc:644] Failed to heartbeat to 127.31.204.254:36891 (0 consecutive failures): Network error: Failed to ping master at 127.31.204.254:36891: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
I20260810 06:23:48.855523 28235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:48.855332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:41895)
0810 06:23:48.855428 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:48.855473 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 06:23:48.856096 28235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:48.855905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:44217)
0810 06:23:48.856009 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:48.856049 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 06:23:48.856254 28230 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 06:23:48.859019 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 28015
I20260810 06:23:49.585501 28238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:49.585017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:46383)
0810 06:23:49.585307 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:49.585375 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":79,"threads_started":1}
I20260810 06:23:49.686300 28239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:49.685891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:48731)
0810 06:23:49.686157 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:49.686207 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":70,"threads_started":1}
I20260810 06:23:49.857141 28240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:49.856747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:35105)
0810 06:23:49.856993 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:49.857046 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":68,"threads_started":1}
I20260810 06:23:49.915422 28241 raft_consensus.cc:482] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:23:49.915611 28241 raft_consensus.cc:504] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } }
I20260810 06:23:49.916823 28241 leader_election.cc:288] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33a74eeb639b4843aa79415f13aa5612 (127.31.204.195:43099), e7ef34e827a94f0785ee8b60691af394 (127.31.204.194:45605)
Aug 10 06:23:49 dist-test-slave-q9h9 krb5kdc[25635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343028, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.194@KRBTEST.COM
Aug 10 06:23:49 dist-test-slave-q9h9 krb5kdc[25635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343028, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.195@KRBTEST.COM
I20260810 06:23:49.933151 28238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:49.916987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:43099 (local address 127.31.204.193:49973)
0810 06:23:49.917164 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:49.917179 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:49.917280 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:49.919498 (+ 2218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:23:49.919504 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:23:49.920082 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:23:49.920862 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:49.920883 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:49.922747 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:49.922750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:49.924272 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:49.924283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:49.925209 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:49.925213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:49.925479 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:49.926201 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:23:49.926214 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:23:49.928372 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:23:49.930734 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:23:49.930738 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:23:49.930749 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:23:49.930990 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:23:49.931328 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:23:49.931332 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:23:49.931334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:23:49.931463 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:23:49.932149 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:23:49.932154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:23:49.932438 (+ 284us) client_negotiation.cc:769] Sending connection context
0810 06:23:49.932647 (+ 209us) client_negotiation.cc:240] Negotiation successful
0810 06:23:49.932944 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":286,"spinlock_wait_cycles":19584}
I20260810 06:23:49.933241 28245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:49.917270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:49973 (local address 127.31.204.195:43099)
0810 06:23:49.917575 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:49.917606 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:49.917656 (+ 50us) server_negotiation.cc:407] Connection header received
0810 06:23:49.918608 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:49.918617 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:49.919072 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:49.919407 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:23:49.921034 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:49.922607 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:49.924385 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:49.925100 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:49.925275 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:49.928582 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:23:49.928588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:23:49.928604 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:23:49.928637 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:23:49.930619 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:49.931080 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:49.931084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:49.931088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:49.931228 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:49.931550 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:49.931553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:49.931555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:49.932037 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:23:49.932162 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:49.932772 (+ 610us) server_negotiation.cc:299] Negotiation successful
0810 06:23:49.933052 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":84,"threads_started":1}
I20260810 06:23:49.933239 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:49.917473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:50755 (local address 127.31.204.194:45605)
0810 06:23:49.917802 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0810 06:23:49.917825 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 06:23:49.917861 (+ 36us) server_negotiation.cc:407] Connection header received
0810 06:23:49.918608 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:23:49.918617 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:23:49.919072 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:23:49.919406 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:23:49.921034 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:49.922607 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:49.924385 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:23:49.925100 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:23:49.925265 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:49.928582 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:23:49.928588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:23:49.928604 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:23:49.928637 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:23:49.930619 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:49.931385 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:49.931389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:49.931393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:49.931500 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:23:49.931745 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:23:49.931748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:23:49.931749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:23:49.932176 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:23:49.932272 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 06:23:49.932772 (+ 500us) server_negotiation.cc:299] Negotiation successful
0810 06:23:49.933052 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":86,"threads_started":1}
I20260810 06:23:49.933156 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:49.917048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:45605 (local address 127.31.204.193:50755)
0810 06:23:49.917507 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:49.917520 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:23:49.917614 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:23:49.919498 (+ 1884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:23:49.919504 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:23:49.920082 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:23:49.920862 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:49.920883 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:49.922744 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:49.922748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:49.924272 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:23:49.924283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:23:49.925209 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:23:49.925213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:23:49.925479 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:23:49.926166 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:23:49.926187 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:23:49.928034 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:23:49.931083 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:23:49.931085 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:23:49.931087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:23:49.931283 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:23:49.931596 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:23:49.931598 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:23:49.931600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:23:49.931668 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:23:49.932276 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:23:49.932278 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:23:49.932438 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 06:23:49.932647 (+ 209us) client_negotiation.cc:240] Negotiation successful
0810 06:23:49.933009 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":433,"spinlock_wait_cycles":38784,"thread_start_us":60,"threads_started":1}
W20260810 06:23:49.935607 27594 leader_election.cc:341] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7ef34e827a94f0785ee8b60691af394 (127.31.204.194:45605): Not found: Tablet not found: f9164f5d6e694a61b68c088f773a4bd1
W20260810 06:23:49.935735 27591 leader_election.cc:341] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33a74eeb639b4843aa79415f13aa5612 (127.31.204.195:43099): Not found: Tablet not found: f9164f5d6e694a61b68c088f773a4bd1
I20260810 06:23:49.935823 27591 leader_election.cc:302] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbd3c9c2e84b481fa51beb02c7c81753; no voters: 33a74eeb639b4843aa79415f13aa5612, e7ef34e827a94f0785ee8b60691af394
I20260810 06:23:49.936062 28241 raft_consensus.cc:2726] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 06:23:50.586607 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:50.586234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:53221)
0810 06:23:50.586462 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:50.586513 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":72,"threads_started":1}
I20260810 06:23:50.687253 28251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:50.686892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:58855)
0810 06:23:50.687120 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:50.687171 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
I20260810 06:23:50.858178 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:50.857783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:51871)
0810 06:23:50.858004 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:50.858089 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":76,"threads_started":1}
I20260810 06:23:51.587620 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:51.587240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:45277)
0810 06:23:51.587472 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:51.587524 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20260810 06:23:51.688220 28260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:51.687874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:37637)
0810 06:23:51.688087 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:51.688136 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":69,"threads_started":1}
I20260810 06:23:51.859197 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:51.858791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:40707)
0810 06:23:51.859033 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:51.859089 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260810 06:23:52.156353 28265 raft_consensus.cc:482] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:23:52.156479 28265 raft_consensus.cc:504] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } }
I20260810 06:23:52.156844 28265 leader_election.cc:288] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33a74eeb639b4843aa79415f13aa5612 (127.31.204.195:43099), e7ef34e827a94f0785ee8b60691af394 (127.31.204.194:45605)
W20260810 06:23:52.157693 27594 leader_election.cc:341] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7ef34e827a94f0785ee8b60691af394 (127.31.204.194:45605): Not found: Tablet not found: f9164f5d6e694a61b68c088f773a4bd1
W20260810 06:23:52.157850 27591 leader_election.cc:341] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33a74eeb639b4843aa79415f13aa5612 (127.31.204.195:43099): Not found: Tablet not found: f9164f5d6e694a61b68c088f773a4bd1
I20260810 06:23:52.157912 27591 leader_election.cc:302] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbd3c9c2e84b481fa51beb02c7c81753; no voters: 33a74eeb639b4843aa79415f13aa5612, e7ef34e827a94f0785ee8b60691af394
I20260810 06:23:52.158087 28265 raft_consensus.cc:2726] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 06:23:52.588707 28267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:52.588287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:38537)
0810 06:23:52.588553 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:52.588606 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":95,"threads_started":1}
I20260810 06:23:52.689224 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:52.688832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:40299)
0810 06:23:52.689082 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:52.689138 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
I20260810 06:23:52.860178 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:52.859820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:49099)
0810 06:23:52.860040 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:52.860092 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":74,"threads_started":1}
I20260810 06:23:53.590060 28274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:53.589551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:51479)
0810 06:23:53.589857 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:53.589919 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":75,"threads_started":1}
I20260810 06:23:53.690389 28275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:53.689930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:51255)
0810 06:23:53.690202 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:53.690274 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":114,"threads_started":1}
I20260810 06:23:53.861234 28276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:53.860854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:60919)
0810 06:23:53.861092 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:53.861143 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
I20260810 06:23:54.105729 28277 raft_consensus.cc:482] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:23:54.105861 28277 raft_consensus.cc:504] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } }
I20260810 06:23:54.106292 28277 leader_election.cc:288] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33a74eeb639b4843aa79415f13aa5612 (127.31.204.195:43099), e7ef34e827a94f0785ee8b60691af394 (127.31.204.194:45605)
W20260810 06:23:54.107041 27594 leader_election.cc:341] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7ef34e827a94f0785ee8b60691af394 (127.31.204.194:45605): Not found: Tablet not found: f9164f5d6e694a61b68c088f773a4bd1
W20260810 06:23:54.107170 27591 leader_election.cc:341] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33a74eeb639b4843aa79415f13aa5612 (127.31.204.195:43099): Not found: Tablet not found: f9164f5d6e694a61b68c088f773a4bd1
I20260810 06:23:54.107259 27591 leader_election.cc:302] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbd3c9c2e84b481fa51beb02c7c81753; no voters: 33a74eeb639b4843aa79415f13aa5612, e7ef34e827a94f0785ee8b60691af394
I20260810 06:23:54.107388 28277 raft_consensus.cc:2726] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 06:23:54.591208 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:54.590786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:45341)
0810 06:23:54.591049 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:54.591106 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
I20260810 06:23:54.691394 28279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:54.691010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:50109)
0810 06:23:54.691251 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:54.691307 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
I20260810 06:23:54.862264 28280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:54.861837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:36215)
0810 06:23:54.862124 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:54.862179 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":85,"threads_started":1}
I20260810 06:23:55.592274 28282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:55.591868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:43065)
0810 06:23:55.592123 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:55.592176 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":79,"threads_started":1}
I20260810 06:23:55.692458 28283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:55.692017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:32917)
0810 06:23:55.692318 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:55.692372 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":82,"threads_started":1}
I20260810 06:23:55.739104 28286 raft_consensus.cc:482] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:23:55.739226 28286 raft_consensus.cc:504] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } }
I20260810 06:23:55.739614 28286 leader_election.cc:288] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33a74eeb639b4843aa79415f13aa5612 (127.31.204.195:43099), e7ef34e827a94f0785ee8b60691af394 (127.31.204.194:45605)
W20260810 06:23:55.740293 27594 leader_election.cc:341] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7ef34e827a94f0785ee8b60691af394 (127.31.204.194:45605): Not found: Tablet not found: f9164f5d6e694a61b68c088f773a4bd1
W20260810 06:23:55.740406 27591 leader_election.cc:341] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33a74eeb639b4843aa79415f13aa5612 (127.31.204.195:43099): Not found: Tablet not found: f9164f5d6e694a61b68c088f773a4bd1
I20260810 06:23:55.740481 27591 leader_election.cc:302] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbd3c9c2e84b481fa51beb02c7c81753; no voters: 33a74eeb639b4843aa79415f13aa5612, e7ef34e827a94f0785ee8b60691af394
I20260810 06:23:55.740625 28286 raft_consensus.cc:2726] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 06:23:55.863283 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:55.862860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:52997)
0810 06:23:55.863132 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:55.863184 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":69,"threads_started":1}
I20260810 06:23:56.593351 28288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:56.592931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:52611)
0810 06:23:56.593201 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:56.593255 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":80,"threads_started":1}
I20260810 06:23:56.693434 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:56.693045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:42585)
0810 06:23:56.693287 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:56.693343 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":69,"threads_started":1}
I20260810 06:23:56.864351 28290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:56.863863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:53251)
0810 06:23:56.864171 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:56.864252 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":87,"threads_started":1}
I20260810 06:23:57.594528 28292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:57.594100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:49937)
0810 06:23:57.594378 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:57.594432 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":88,"threads_started":1}
I20260810 06:23:57.694540 28296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:57.694133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:39249)
0810 06:23:57.694389 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:57.694443 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
I20260810 06:23:57.865443 28297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:57.865025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:45913)
0810 06:23:57.865272 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:57.865327 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20260810 06:23:58.433717 28298 raft_consensus.cc:482] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:23:58.433841 28298 raft_consensus.cc:504] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a74eeb639b4843aa79415f13aa5612" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 43099 } } peers { permanent_uuid: "e7ef34e827a94f0785ee8b60691af394" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 45605 } } peers { permanent_uuid: "bbd3c9c2e84b481fa51beb02c7c81753" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38105 } }
I20260810 06:23:58.434268 28298 leader_election.cc:288] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33a74eeb639b4843aa79415f13aa5612 (127.31.204.195:43099), e7ef34e827a94f0785ee8b60691af394 (127.31.204.194:45605)
W20260810 06:23:58.435002 27591 leader_election.cc:341] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33a74eeb639b4843aa79415f13aa5612 (127.31.204.195:43099): Not found: Tablet not found: f9164f5d6e694a61b68c088f773a4bd1
W20260810 06:23:58.435107 27594 leader_election.cc:341] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7ef34e827a94f0785ee8b60691af394 (127.31.204.194:45605): Not found: Tablet not found: f9164f5d6e694a61b68c088f773a4bd1
I20260810 06:23:58.435171 27594 leader_election.cc:302] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbd3c9c2e84b481fa51beb02c7c81753; no voters: 33a74eeb639b4843aa79415f13aa5612, e7ef34e827a94f0785ee8b60691af394
I20260810 06:23:58.435345 28298 raft_consensus.cc:2726] T f9164f5d6e694a61b68c088f773a4bd1 P bbd3c9c2e84b481fa51beb02c7c81753 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 06:23:58.595635 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:58.595253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:37549)
0810 06:23:58.595489 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:58.595541 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260810 06:23:58.695566 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:58.695188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:46597)
0810 06:23:58.695420 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:58.695473 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":79,"threads_started":1}
I20260810 06:23:58.866477 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:58.866112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:37969)
0810 06:23:58.866334 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:58.866384 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
I20260810 06:23:59.596760 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:59.596356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:42925)
0810 06:23:59.596605 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:59.596660 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
I20260810 06:23:59.696599 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:59.696219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:54937)
0810 06:23:59.696452 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:59.696506 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":81,"threads_started":1}
I20260810 06:23:59.867467 28304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:23:59.867085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:33295)
0810 06:23:59.867326 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 06:23:59.867380 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":87,"threads_started":1}
I20260810 06:24:00.597847 28305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:00.597438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:37361)
0810 06:24:00.597696 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:00.597745 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
I20260810 06:24:00.697739 28306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:00.697308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:44503)
0810 06:24:00.697588 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:00.697645 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
I20260810 06:24:00.868458 28308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:00.868072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:38917)
0810 06:24:00.868312 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:00.868366 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":73,"threads_started":1}
I20260810 06:24:01.599030 28309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:01.598601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:56601)
0810 06:24:01.598867 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:01.598923 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":95,"threads_started":1}
I20260810 06:24:01.698879 28311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:01.698460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:36213)
0810 06:24:01.698725 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:01.698791 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":90,"threads_started":1}
I20260810 06:24:01.869582 28312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:01.869119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:44485)
0810 06:24:01.869407 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:01.869482 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":97,"threads_started":1}
I20260810 06:24:02.600096 28313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:02.599698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:48603)
0810 06:24:02.599935 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:02.599994 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":69,"threads_started":1}
I20260810 06:24:02.699978 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:02.699544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:43737)
0810 06:24:02.699786 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:02.699850 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":77,"threads_started":1}
I20260810 06:24:02.870638 28316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:02.870243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:39449)
0810 06:24:02.870487 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:02.870541 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":77,"threads_started":1}
I20260810 06:24:03.601415 28319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:03.600998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.193:52725)
0810 06:24:03.601261 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:03.601317 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260810 06:24:03.701269 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:03.700860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.194:56837)
0810 06:24:03.701112 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:03.701164 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":91,"threads_started":1}
I20260810 06:24:03.859499 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 27505
I20260810 06:24:03.872367 28322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:03.871371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36891 (local address 127.31.204.195:55403)
0810 06:24:03.872214 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:03.872276 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.254:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":545,"threads_started":1}
I20260810 06:24:03.877504 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 27778
I20260810 06:24:03.887718 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 28015
2026-08-10T06:24:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59658 ms)
[----------] 30 tests from SecurityITest (240187 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:03 dist-test-slave-q9h9 krb5kdc[28325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:03 dist-test-slave-q9h9 krb5kdc[28325](info): set up 2 sockets
Aug 10 06:24:03 dist-test-slave-q9h9 krb5kdc[28325](info): commencing operation
krb5kdc: starting...
W20260810 06:24:05.048615 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:05 dist-test-slave-q9h9 krb5kdc[28325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:24:05.158262 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:45981
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:45981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:05.265755 28353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:05.265995 28353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:05.266072 28353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:05.270248 28353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:05.270301 28353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:05.270327 28353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:05.270350 28353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:05.270380 28353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:05.275465 28353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:45981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:45981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:05.276473 28353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:05.277470 28353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:05.286240 28372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:05.286291 28360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:05.286329 28362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:05.286616 28353 server_base.cc:1034] running on GCE node
I20260810 06:24:05.287171 28353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:05.288236 28353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:05.289393 28353 hybrid_clock.cc:648] HybridClock initialized: now 1786343045289379 us; error 25 us; skew 500 ppm
Aug 10 06:24:05 dist-test-slave-q9h9 krb5kdc[28325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343045, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:05.292014 28353 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:24:05.292874 28353 webserver.cc:460] Webserver started at http://127.31.204.254:44951/ using document root <none> and password file <none>
I20260810 06:24:05.293397 28353 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:05.293464 28353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:05.293669 28353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:05.295696 28353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "2ab4ac108548481281cd608d693d6765"
format_stamp: "Formatted at 2026-08-10 06:24:05 on dist-test-slave-q9h9"
server_key: "eedad4929dba87e3e74749ef7e44d931"
server_key_iv: "9b9f9454ba5c6b5d0a1fe6b5500d4fdd"
server_key_version: "encryptionkey@0"
I20260810 06:24:05.296238 28353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "2ab4ac108548481281cd608d693d6765"
format_stamp: "Formatted at 2026-08-10 06:24:05 on dist-test-slave-q9h9"
server_key: "eedad4929dba87e3e74749ef7e44d931"
server_key_iv: "9b9f9454ba5c6b5d0a1fe6b5500d4fdd"
server_key_version: "encryptionkey@0"
I20260810 06:24:05.299782 28353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 06:24:05.301954 28387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:05.303103 28353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:05.303217 28353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "2ab4ac108548481281cd608d693d6765"
format_stamp: "Formatted at 2026-08-10 06:24:05 on dist-test-slave-q9h9"
server_key: "eedad4929dba87e3e74749ef7e44d931"
server_key_iv: "9b9f9454ba5c6b5d0a1fe6b5500d4fdd"
server_key_version: "encryptionkey@0"
I20260810 06:24:05.303309 28353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:05.329175 28353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:05.331795 28353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:05.331971 28353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:05.338572 28353 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:45981
I20260810 06:24:05.338615 28465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:45981 every 8 connection(s)
I20260810 06:24:05.339478 28353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:05.342135 28466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:05.343576 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 28353
I20260810 06:24:05.343703 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:05.343984 32563 external_mini_cluster.cc:1428] Setting key c4f0feb8b790adc9cd6d63c5546ef31b
I20260810 06:24:05.348042 28466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765: Bootstrap starting.
I20260810 06:24:05.350490 28466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:05.351334 28466 log.cc:824] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765: Log is configured to *not* fsync() on all Append() calls
Aug 10 06:24:05 dist-test-slave-q9h9 krb5kdc[28325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:24:05.353276 28466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765: No bootstrap required, opened a new log
I20260810 06:24:05.357654 28466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ab4ac108548481281cd608d693d6765" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45981 } }
I20260810 06:24:05.357920 28466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:05.357977 28466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab4ac108548481281cd608d693d6765, State: Initialized, Role: FOLLOWER
I20260810 06:24:05.357992 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:05.345249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.31.204.254:45981)
0810 06:24:05.345646 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:05.345661 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:05.345698 (+ 37us) server_negotiation.cc:407] Connection header received
0810 06:24:05.346357 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:05.346377 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:05.346696 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:05.346965 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:05.347831 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:05.348836 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:05.349985 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:05.350484 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:05.350612 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:05.353162 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:05.353168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:05.353183 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:05.353213 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:05.355038 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:05.355449 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:05.355455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:05.355461 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:05.355607 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:05.355838 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:05.355841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:05.355843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:05.356320 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:05.356505 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:05.356733 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 06:24:05.357037 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":107,"threads_started":1}
I20260810 06:24:05.358453 28466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ab4ac108548481281cd608d693d6765" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45981 } }
I20260810 06:24:05.358610 28466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:05.358673 28466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:05.358764 28466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:05.359601 28466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ab4ac108548481281cd608d693d6765" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45981 } }
I20260810 06:24:05.359908 28466 leader_election.cc:302] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ab4ac108548481281cd608d693d6765; no voters:
I20260810 06:24:05.360255 28466 leader_election.cc:288] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:05.360338 28474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:05.360535 28474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765 [term 1 LEADER]: Becoming Leader. State: Replica: 2ab4ac108548481281cd608d693d6765, State: Running, Role: LEADER
I20260810 06:24:05.360803 28474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ab4ac108548481281cd608d693d6765" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45981 } }
I20260810 06:24:05.361352 28466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:05.361894 28474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ab4ac108548481281cd608d693d6765. Latest consensus state: current_term: 1 leader_uuid: "2ab4ac108548481281cd608d693d6765" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ab4ac108548481281cd608d693d6765" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45981 } } }
I20260810 06:24:05.362486 28476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ab4ac108548481281cd608d693d6765" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ab4ac108548481281cd608d693d6765" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45981 } } }
I20260810 06:24:05.362587 28474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:05.362656 28476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:05.363062 28488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:05.365860 28488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:05.371323 28488 catalog_manager.cc:1347] Generated new cluster ID: 653a17e2f69a4fbbbc5689134d4290e4
I20260810 06:24:05.371436 28488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:05.387919 28488 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:05.388773 28488 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:05.395900 28488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765: Generated new TSK 0
I20260810 06:24:05.396494 28488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
I20260810 06:24:05.432832 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:45981
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:05.570556 28507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:05.570794 28507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:05.570835 28507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:05.574712 28507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:05.574761 28507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:05.574833 28507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:24:05.579887 28507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:45981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:05.580835 28507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:05.581776 28507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:24:05.591373 28507 server_base.cc:1034] running on GCE node
W20260810 06:24:05.591343 28518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:05.591302 28520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:05.591311 28517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:05.591997 28507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:05.592579 28507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:05.593741 28507 hybrid_clock.cc:648] HybridClock initialized: now 1786343045593724 us; error 28 us; skew 500 ppm
Aug 10 06:24:05 dist-test-slave-q9h9 krb5kdc[28325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343045, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:05.596346 28507 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:24:05.597162 28507 webserver.cc:460] Webserver started at http://127.31.204.193:40675/ using document root <none> and password file <none>
I20260810 06:24:05.597688 28507 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:05.597754 28507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:05.597965 28507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:05.600021 28507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "9598e60340f742318ed8b5c447c653d5"
format_stamp: "Formatted at 2026-08-10 06:24:05 on dist-test-slave-q9h9"
server_key: "377231771097584ef20fa3b7839db4d4"
server_key_iv: "3a35e2b3223faff0785dcd8d19845d1a"
server_key_version: "encryptionkey@0"
I20260810 06:24:05.600558 28507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "9598e60340f742318ed8b5c447c653d5"
format_stamp: "Formatted at 2026-08-10 06:24:05 on dist-test-slave-q9h9"
server_key: "377231771097584ef20fa3b7839db4d4"
server_key_iv: "3a35e2b3223faff0785dcd8d19845d1a"
server_key_version: "encryptionkey@0"
I20260810 06:24:05.604116 28507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:24:05.606408 28535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:05.607573 28507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:24:05.607694 28507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "9598e60340f742318ed8b5c447c653d5"
format_stamp: "Formatted at 2026-08-10 06:24:05 on dist-test-slave-q9h9"
server_key: "377231771097584ef20fa3b7839db4d4"
server_key_iv: "3a35e2b3223faff0785dcd8d19845d1a"
server_key_version: "encryptionkey@0"
I20260810 06:24:05.607805 28507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:05.618467 28507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:05.620967 28507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:05.621133 28507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:05.621688 28507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:05.622536 28507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:05.622586 28507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:05.622646 28507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:05.622674 28507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:05.633912 28507 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:43233
I20260810 06:24:05.633970 28719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:43233 every 8 connection(s)
I20260810 06:24:05.634867 28507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:24:05.639986 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 28507
I20260810 06:24:05.640115 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:24:05.640432 32563 external_mini_cluster.cc:1428] Setting key 1d581b5d3abd7264d825899da9b79efe
Aug 10 06:24:05 dist-test-slave-q9h9 krb5kdc[28325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343045, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:24:05.648334 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:05.636643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:47715 (local address 127.31.204.254:45981)
0810 06:24:05.636801 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:05.636806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:05.637668 (+ 862us) server_negotiation.cc:407] Connection header received
0810 06:24:05.638999 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:05.639003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:05.639050 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:05.639123 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:05.640296 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:05.640987 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:05.642231 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:05.642533 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:05.642636 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:05.644916 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:05.644919 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:05.644920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:05.644945 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:05.646233 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:05.646841 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:05.646844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:05.646846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:05.646938 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:05.647310 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:05.647312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:05.647313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:05.647476 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:05.647547 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:05.648120 (+ 573us) server_negotiation.cc:299] Negotiation successful
0810 06:24:05.648215 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 06:24:05.649523 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:05.637046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:45981 (local address 127.31.204.193:47715)
0810 06:24:05.637517 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:05.637559 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:05.638804 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:05.639272 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:05.639278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:05.639613 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:05.640091 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:05.640109 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:05.641108 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:05.641111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:05.642112 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:05.642124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:05.642667 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:05.642669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:05.642802 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:05.643342 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:05.643357 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:05.644763 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:05.646342 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:05.646349 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:05.646362 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:05.646750 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:05.647022 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:05.647026 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:05.647028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:05.647227 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:05.647575 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:05.647581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:05.647890 (+ 309us) client_negotiation.cc:769] Sending connection context
0810 06:24:05.648105 (+ 215us) client_negotiation.cc:240] Negotiation successful
0810 06:24:05.648477 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":124,"threads_started":1}
I20260810 06:24:05.650847 28721 heartbeater.cc:347] Connected to a master server at 127.31.204.254:45981
I20260810 06:24:05.651108 28721 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.31.204.194@KRBTEST.COM; defaulting to no policy
I20260810 06:24:05.651621 28721 heartbeater.cc:511] Master 127.31.204.254:45981 requested a full tablet report, sending...
I20260810 06:24:05.653092 28412 ts_manager.cc:194] Registered new tserver with Master: 9598e60340f742318ed8b5c447c653d5 (127.31.204.193:43233)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.194@KRBTEST.COM" created.
I20260810 06:24:05.654407 28412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:47715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
I20260810 06:24:05.667097 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:45981
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:05.775151 28730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:05.775370 28730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:05.775409 28730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:05.779271 28730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:05.779317 28730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:05.779371 28730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:24:05.784323 28730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:45981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:05.785184 28730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:05.786118 28730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:05.794934 28735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:05.794935 28741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:05.794930 28738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:05.795291 28730 server_base.cc:1034] running on GCE node
I20260810 06:24:05.795632 28730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:05.796224 28730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:05.797376 28730 hybrid_clock.cc:648] HybridClock initialized: now 1786343045797362 us; error 29 us; skew 500 ppm
Aug 10 06:24:05 dist-test-slave-q9h9 krb5kdc[28325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343045, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:05.799922 28730 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:24:05.800736 28730 webserver.cc:460] Webserver started at http://127.31.204.194:44127/ using document root <none> and password file <none>
I20260810 06:24:05.801240 28730 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:05.801304 28730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:05.801476 28730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:05.803411 28730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "aac54102787c4dd1aebed44d044bd9e5"
format_stamp: "Formatted at 2026-08-10 06:24:05 on dist-test-slave-q9h9"
server_key: "c5f7912f2ea825898e42993b74d1b2bb"
server_key_iv: "2f8fe1fcaff666d82551d069a810178f"
server_key_version: "encryptionkey@0"
I20260810 06:24:05.803920 28730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "aac54102787c4dd1aebed44d044bd9e5"
format_stamp: "Formatted at 2026-08-10 06:24:05 on dist-test-slave-q9h9"
server_key: "c5f7912f2ea825898e42993b74d1b2bb"
server_key_iv: "2f8fe1fcaff666d82551d069a810178f"
server_key_version: "encryptionkey@0"
I20260810 06:24:05.807384 28730 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:24:05.809652 28750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:05.810871 28730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:24:05.810977 28730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "aac54102787c4dd1aebed44d044bd9e5"
format_stamp: "Formatted at 2026-08-10 06:24:05 on dist-test-slave-q9h9"
server_key: "c5f7912f2ea825898e42993b74d1b2bb"
server_key_iv: "2f8fe1fcaff666d82551d069a810178f"
server_key_version: "encryptionkey@0"
I20260810 06:24:05.811062 28730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:05.839927 28730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:05.842916 28730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:05.843083 28730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:05.843629 28730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:05.844404 28730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:05.844449 28730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:05.844501 28730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:05.844527 28730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:05.852795 28730 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:41499
I20260810 06:24:05.852830 28951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:41499 every 8 connection(s)
I20260810 06:24:05.853621 28730 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
Aug 10 06:24:05 dist-test-slave-q9h9 krb5kdc[28325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343045, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:24:05.863189 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 28730
I20260810 06:24:05.863312 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:24:05.863641 32563 external_mini_cluster.cc:1428] Setting key efddbb0504820fa3a468b3115efb9891
I20260810 06:24:05.866534 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:05.855252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:57479 (local address 127.31.204.254:45981)
0810 06:24:05.855368 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:05.855372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:05.856114 (+ 742us) server_negotiation.cc:407] Connection header received
0810 06:24:05.857368 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:05.857372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:05.857432 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:05.857504 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:05.858649 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:05.859158 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:05.860275 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:05.860490 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:05.860579 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:05.862953 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:05.862957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:05.862960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:05.862992 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:05.864434 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:05.865007 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:05.865010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:05.865011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:05.865103 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:05.865456 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:05.865460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:05.865462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:05.865682 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:05.865782 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:05.866259 (+ 477us) server_negotiation.cc:299] Negotiation successful
0810 06:24:05.866377 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 06:24:05.867306 28958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:05.855558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:45981 (local address 127.31.204.194:57479)
0810 06:24:05.855969 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:05.856010 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:05.857161 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:05.857673 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:05.857680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:05.858007 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:05.858501 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:05.858514 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:05.859279 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:05.859281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:05.860142 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:05.860152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:05.860627 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:05.860631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:05.860794 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:05.861284 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:05.861300 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:05.862770 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:05.864540 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:05.864545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:05.864558 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:05.864895 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:05.865224 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:05.865226 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:05.865228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:05.865361 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:05.865774 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:05.865778 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:05.865985 (+ 207us) client_negotiation.cc:769] Sending connection context
0810 06:24:05.866173 (+ 188us) client_negotiation.cc:240] Negotiation successful
0810 06:24:05.866477 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":98,"threads_started":1}
I20260810 06:24:05.868471 28955 heartbeater.cc:347] Connected to a master server at 127.31.204.254:45981
I20260810 06:24:05.868695 28955 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:05.869197 28955 heartbeater.cc:511] Master 127.31.204.254:45981 requested a full tablet report, sending...
I20260810 06:24:05.870282 28412 ts_manager.cc:194] Registered new tserver with Master: aac54102787c4dd1aebed44d044bd9e5 (127.31.204.194:41499)
I20260810 06:24:05.871034 28412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:57479
WARNING: no policy specified for kudu/127.31.204.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
I20260810 06:24:05.890036 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:45981
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:05.996294 28963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:05.996518 28963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:05.996557 28963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:06.000455 28963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:06.000504 28963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:06.000572 28963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:24:06.005549 28963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:45981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:06.006491 28963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:06.007431 28963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:06.016355 28975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:06.016399 28980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:06.016417 28963 server_base.cc:1034] running on GCE node
W20260810 06:24:06.016363 28972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:06.016995 28963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:06.017556 28963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:06.018703 28963 hybrid_clock.cc:648] HybridClock initialized: now 1786343046018686 us; error 31 us; skew 500 ppm
Aug 10 06:24:06 dist-test-slave-q9h9 krb5kdc[28325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343046, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:06.021250 28963 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:24:06.022051 28963 webserver.cc:460] Webserver started at http://127.31.204.195:44531/ using document root <none> and password file <none>
I20260810 06:24:06.022552 28963 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:06.022617 28963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:06.022801 28963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:06.024709 28963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "c283d6f8d1f9460db1e8bbdf76e1c012"
format_stamp: "Formatted at 2026-08-10 06:24:06 on dist-test-slave-q9h9"
server_key: "58183d85eb530072504a0b2ee88afcd1"
server_key_iv: "bdc6e953944e98364c3838d55b777d01"
server_key_version: "encryptionkey@0"
I20260810 06:24:06.025235 28963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "c283d6f8d1f9460db1e8bbdf76e1c012"
format_stamp: "Formatted at 2026-08-10 06:24:06 on dist-test-slave-q9h9"
server_key: "58183d85eb530072504a0b2ee88afcd1"
server_key_iv: "bdc6e953944e98364c3838d55b777d01"
server_key_version: "encryptionkey@0"
I20260810 06:24:06.028805 28963 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 06:24:06.031087 28988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:06.032187 28963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:24:06.032292 28963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "c283d6f8d1f9460db1e8bbdf76e1c012"
format_stamp: "Formatted at 2026-08-10 06:24:06 on dist-test-slave-q9h9"
server_key: "58183d85eb530072504a0b2ee88afcd1"
server_key_iv: "bdc6e953944e98364c3838d55b777d01"
server_key_version: "encryptionkey@0"
I20260810 06:24:06.032379 28963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:06.047480 28963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:06.050423 28963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:06.050591 28963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:06.051144 28963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:06.051940 28963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:06.051988 28963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:06.052039 28963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:06.052069 28963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:06.060482 28963 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:34667
I20260810 06:24:06.060504 29165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:34667 every 8 connection(s)
I20260810 06:24:06.061306 28963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:24:06.065375 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 28963
I20260810 06:24:06.065485 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:24:06.065809 32563 external_mini_cluster.cc:1428] Setting key 723217afc1792a587a602104c2a0d6fb
Aug 10 06:24:06 dist-test-slave-q9h9 krb5kdc[28325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343046, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:24:06.074004 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:06.063128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:58201 (local address 127.31.204.254:45981)
0810 06:24:06.063269 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:06.063272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:06.064083 (+ 811us) server_negotiation.cc:407] Connection header received
0810 06:24:06.065336 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:06.065339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:06.065396 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:06.065480 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:06.066870 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.067468 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.068668 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.068888 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.068966 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:06.071276 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:06.071278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:06.071280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:06.071305 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:06.072419 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:06.072854 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:06.072856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:06.072857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:06.072920 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:06.073160 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:06.073162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:06.073163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:06.073329 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:06.073397 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:06.073794 (+ 397us) server_negotiation.cc:299] Negotiation successful
0810 06:24:06.073889 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 06:24:06.074797 29171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:06.063482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:45981 (local address 127.31.204.195:58201)
0810 06:24:06.063940 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:06.063979 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:06.065125 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:06.065698 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:06.065708 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:06.066190 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:06.066721 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:06.066738 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.067591 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.067594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:06.068562 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:06.068572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.068997 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.068999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:06.069134 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:06.069644 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:06.069660 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:06.071121 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:06.072513 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:06.072518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:06.072528 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:06.072779 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:06.072977 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:06.072979 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:06.072980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:06.073098 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:06.073397 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:06.073402 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:06.073612 (+ 210us) client_negotiation.cc:769] Sending connection context
0810 06:24:06.073764 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 06:24:06.073997 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":112,"threads_started":1}
I20260810 06:24:06.075786 29166 heartbeater.cc:347] Connected to a master server at 127.31.204.254:45981
I20260810 06:24:06.075999 29166 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:06.076448 29166 heartbeater.cc:511] Master 127.31.204.254:45981 requested a full tablet report, sending...
I20260810 06:24:06.077378 28412 ts_manager.cc:194] Registered new tserver with Master: c283d6f8d1f9460db1e8bbdf76e1c012 (127.31.204.195:34667)
I20260810 06:24:06.077917 28412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:58201
I20260810 06:24:06.079540 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:24:06.087862 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:06.081475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35492 (local address 127.31.204.254:45981)
0810 06:24:06.081588 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:06.081591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:06.081750 (+ 159us) server_negotiation.cc:407] Connection header received
0810 06:24:06.081846 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:06.081848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:06.081881 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:06.081939 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:06.082590 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.083129 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.083889 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.084087 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.084150 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:06.085307 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:06.085309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:06.085311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:06.085332 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:06.086572 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:06.086903 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:06.086905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:06.086906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:06.086970 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:06.087150 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:06.087152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:06.087153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:06.087336 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:06.087434 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:06.087653 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 06:24:06.087744 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:24:06.090339 28412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:24:06.093202 28412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:24:06.106590 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:06.101014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59474 (local address 127.31.204.194:41499)
0810 06:24:06.102127 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:06.102140 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:06.102174 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:24:06.102250 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:06.102255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:06.102457 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:06.102600 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:06.102838 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.103736 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.105119 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.105758 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.105878 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:06.106076 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:06.106348 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 06:24:06.106447 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":984,"thread_start_us":110,"threads_started":1}
I20260810 06:24:06.106598 29189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:06.100878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:41499 (local address 127.0.0.1:59474)
0810 06:24:06.101940 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:06.101956 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:06.102077 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:06.102594 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:06.102598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:06.102614 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:06.102711 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:06.102716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.103844 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.103848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:06.104996 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:06.105007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.106045 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.106049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:06.106157 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:06.106196 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 06:24:06.106302 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 06:24:06.106413 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":42,"thread_start_us":79,"threads_started":1}
I20260810 06:24:06.107146 29188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:06.101010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:34667 (local address 127.0.0.1:43064)
0810 06:24:06.101890 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:06.101940 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:06.102076 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:06.103319 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:06.103322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:06.103341 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:06.103434 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:06.103439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.104723 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.104727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:06.105739 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:06.105747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.106770 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.106774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:06.106857 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:06.106879 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:24:06.106946 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 06:24:06.107016 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260810 06:24:06.107333 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:06.101521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:43233 (local address 127.0.0.1:36280)
0810 06:24:06.102274 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:06.102285 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:06.102349 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:06.102787 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:06.102790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:06.102808 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:06.102894 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:06.102900 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.104272 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.104275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:06.105435 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:06.105445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.106866 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.106872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:06.106975 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:06.107000 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 06:24:06.107069 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 06:24:06.107133 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":4,"thread_start_us":91,"threads_started":1}
I20260810 06:24:06.107520 29193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:06.101664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36280 (local address 127.31.204.193:43233)
0810 06:24:06.102386 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:06.102396 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:06.102418 (+ 22us) server_negotiation.cc:407] Connection header received
0810 06:24:06.102487 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:06.102492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:06.102659 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:06.102803 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:06.103002 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.104095 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.105560 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.106322 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.106482 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:06.106600 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:06.107240 (+ 640us) server_negotiation.cc:299] Negotiation successful
0810 06:24:06.107335 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":108,"threads_started":1}
I20260810 06:24:06.107831 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:06.101584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43064 (local address 127.31.204.195:34667)
0810 06:24:06.102916 (+ 1332us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:06.102927 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:06.102947 (+ 20us) server_negotiation.cc:407] Connection header received
0810 06:24:06.103012 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:06.103016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:06.103187 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:06.103328 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:06.103535 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.104569 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.105849 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.106663 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.106775 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:06.106890 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:06.106969 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 06:24:06.107692 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1234,"thread_start_us":174,"threads_started":1}
I20260810 06:24:06.109923 28839 tablet_service.cc:1467] Processing CreateTablet for tablet 16c0c2d300fe4409b9084eef514bc45d (DEFAULT_TABLE table=test-table [id=c121284c54fb41d09c8bdea344d71872]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:24:06.110595 28608 tablet_service.cc:1467] Processing CreateTablet for tablet 16c0c2d300fe4409b9084eef514bc45d (DEFAULT_TABLE table=test-table [id=c121284c54fb41d09c8bdea344d71872]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:24:06.111089 28839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c0c2d300fe4409b9084eef514bc45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:06.111618 28608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c0c2d300fe4409b9084eef514bc45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:06.111608 29059 tablet_service.cc:1467] Processing CreateTablet for tablet 16c0c2d300fe4409b9084eef514bc45d (DEFAULT_TABLE table=test-table [id=c121284c54fb41d09c8bdea344d71872]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:24:06.112597 29059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c0c2d300fe4409b9084eef514bc45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:06.118368 29194 tablet_bootstrap.cc:492] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012: Bootstrap starting.
I20260810 06:24:06.119418 29196 tablet_bootstrap.cc:492] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5: Bootstrap starting.
I20260810 06:24:06.119588 29195 tablet_bootstrap.cc:492] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5: Bootstrap starting.
I20260810 06:24:06.120901 29194 tablet_bootstrap.cc:654] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:06.121272 29195 tablet_bootstrap.cc:654] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:06.121511 29196 tablet_bootstrap.cc:654] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:06.122277 29194 log.cc:824] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:06.122398 29195 log.cc:824] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:06.122742 29196 log.cc:824] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:06.124152 29195 tablet_bootstrap.cc:492] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5: No bootstrap required, opened a new log
I20260810 06:24:06.124244 29194 tablet_bootstrap.cc:492] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012: No bootstrap required, opened a new log
I20260810 06:24:06.124348 29195 ts_tablet_manager.cc:1397] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 06:24:06.124428 29194 ts_tablet_manager.cc:1397] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 06:24:06.124537 29196 tablet_bootstrap.cc:492] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5: No bootstrap required, opened a new log
I20260810 06:24:06.124701 29196 ts_tablet_manager.cc:1397] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:24:06.127216 29195 raft_consensus.cc:348] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac54102787c4dd1aebed44d044bd9e5" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41499 } } peers { permanent_uuid: "c283d6f8d1f9460db1e8bbdf76e1c012" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 34667 } } peers { permanent_uuid: "9598e60340f742318ed8b5c447c653d5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43233 } }
I20260810 06:24:06.127418 29195 raft_consensus.cc:374] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:06.127497 29195 raft_consensus.cc:729] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac54102787c4dd1aebed44d044bd9e5, State: Initialized, Role: FOLLOWER
I20260810 06:24:06.127905 29195 consensus_queue.cc:260] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac54102787c4dd1aebed44d044bd9e5" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41499 } } peers { permanent_uuid: "c283d6f8d1f9460db1e8bbdf76e1c012" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 34667 } } peers { permanent_uuid: "9598e60340f742318ed8b5c447c653d5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43233 } }
I20260810 06:24:06.128121 29194 raft_consensus.cc:348] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac54102787c4dd1aebed44d044bd9e5" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41499 } } peers { permanent_uuid: "c283d6f8d1f9460db1e8bbdf76e1c012" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 34667 } } peers { permanent_uuid: "9598e60340f742318ed8b5c447c653d5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43233 } }
I20260810 06:24:06.128121 29196 raft_consensus.cc:348] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac54102787c4dd1aebed44d044bd9e5" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41499 } } peers { permanent_uuid: "c283d6f8d1f9460db1e8bbdf76e1c012" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 34667 } } peers { permanent_uuid: "9598e60340f742318ed8b5c447c653d5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43233 } }
I20260810 06:24:06.128404 29194 raft_consensus.cc:374] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:06.128458 29194 raft_consensus.cc:729] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c283d6f8d1f9460db1e8bbdf76e1c012, State: Initialized, Role: FOLLOWER
I20260810 06:24:06.128531 29196 raft_consensus.cc:374] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:06.128618 29196 raft_consensus.cc:729] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9598e60340f742318ed8b5c447c653d5, State: Initialized, Role: FOLLOWER
I20260810 06:24:06.128659 28955 heartbeater.cc:503] Master 127.31.204.254:45981 was elected leader, sending a full tablet report...
I20260810 06:24:06.128865 29195 ts_tablet_manager.cc:1428] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260810 06:24:06.128881 29194 consensus_queue.cc:260] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac54102787c4dd1aebed44d044bd9e5" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41499 } } peers { permanent_uuid: "c283d6f8d1f9460db1e8bbdf76e1c012" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 34667 } } peers { permanent_uuid: "9598e60340f742318ed8b5c447c653d5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43233 } }
I20260810 06:24:06.129503 29166 heartbeater.cc:503] Master 127.31.204.254:45981 was elected leader, sending a full tablet report...
I20260810 06:24:06.129822 29194 ts_tablet_manager.cc:1428] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260810 06:24:06.129822 29196 consensus_queue.cc:260] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac54102787c4dd1aebed44d044bd9e5" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41499 } } peers { permanent_uuid: "c283d6f8d1f9460db1e8bbdf76e1c012" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 34667 } } peers { permanent_uuid: "9598e60340f742318ed8b5c447c653d5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43233 } }
I20260810 06:24:06.130471 28721 heartbeater.cc:503] Master 127.31.204.254:45981 was elected leader, sending a full tablet report...
I20260810 06:24:06.132444 29196 ts_tablet_manager.cc:1428] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260810 06:24:06.135887 28722 tablet.cc:2369] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:24:06.312237 29168 tablet.cc:2369] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:24:06.343415 29206 raft_consensus.cc:482] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:24:06.343600 29206 raft_consensus.cc:504] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac54102787c4dd1aebed44d044bd9e5" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41499 } } peers { permanent_uuid: "c283d6f8d1f9460db1e8bbdf76e1c012" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 34667 } } peers { permanent_uuid: "9598e60340f742318ed8b5c447c653d5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43233 } }
I20260810 06:24:06.344732 29206 leader_election.cc:288] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aac54102787c4dd1aebed44d044bd9e5 (127.31.204.194:41499), c283d6f8d1f9460db1e8bbdf76e1c012 (127.31.204.195:34667)
I20260810 06:24:06.348930 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:06.344918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:41499 (local address 127.31.204.193:33787)
0810 06:24:06.345247 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:06.345260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:06.345402 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:06.345811 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:06.345814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:06.345832 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:06.345971 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:06.345976 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.347025 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.347028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:06.347893 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:06.347903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.348529 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.348532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:06.348625 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:06.348650 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 06:24:06.348712 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 06:24:06.348776 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":36,"spinlock_wait_cycles":28928,"thread_start_us":103,"threads_started":1}
I20260810 06:24:06.348937 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:06.345110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:33787 (local address 127.31.204.194:41499)
0810 06:24:06.345231 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:06.345236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:06.345323 (+ 87us) server_negotiation.cc:407] Connection header received
0810 06:24:06.345528 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:06.345531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:06.345593 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:06.345643 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:06.346093 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.346872 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.348017 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.348425 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.348542 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:06.348594 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:06.348730 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 06:24:06.348803 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:24:06.349633 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c0c2d300fe4409b9084eef514bc45d" candidate_uuid: "9598e60340f742318ed8b5c447c653d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aac54102787c4dd1aebed44d044bd9e5" is_pre_election: true
I20260810 06:24:06.349797 29208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:06.344978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:34667 (local address 127.31.204.193:48497)
0810 06:24:06.345461 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:06.345475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:06.345580 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:06.345833 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:06.345836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:06.345879 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:06.346034 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:06.346040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.346990 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.346993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:06.348301 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:06.348312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.349407 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.349411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:06.349508 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:06.349531 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 06:24:06.349600 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 06:24:06.349659 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":54,"spinlock_wait_cycles":1280,"thread_start_us":102,"threads_started":1}
I20260810 06:24:06.349820 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:06.345369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:48497 (local address 127.31.204.195:34667)
0810 06:24:06.345483 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:06.345487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:06.345556 (+ 69us) server_negotiation.cc:407] Connection header received
0810 06:24:06.345679 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:06.345682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:06.345738 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:06.345810 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:06.346180 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.346887 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.348428 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.348943 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.349267 (+ 324us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:06.349306 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:06.349634 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 06:24:06.349713 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:24:06.349948 28874 raft_consensus.cc:2454] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9598e60340f742318ed8b5c447c653d5 in term 0.
I20260810 06:24:06.350436 28539 leader_election.cc:302] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9598e60340f742318ed8b5c447c653d5, aac54102787c4dd1aebed44d044bd9e5; no voters:
I20260810 06:24:06.350512 29093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c0c2d300fe4409b9084eef514bc45d" candidate_uuid: "9598e60340f742318ed8b5c447c653d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c283d6f8d1f9460db1e8bbdf76e1c012" is_pre_election: true
I20260810 06:24:06.350708 29206 raft_consensus.cc:2781] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:24:06.350798 29206 raft_consensus.cc:482] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:24:06.350778 29093 raft_consensus.cc:2454] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9598e60340f742318ed8b5c447c653d5 in term 0.
I20260810 06:24:06.350841 29206 raft_consensus.cc:3037] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:06.351827 29206 raft_consensus.cc:504] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac54102787c4dd1aebed44d044bd9e5" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41499 } } peers { permanent_uuid: "c283d6f8d1f9460db1e8bbdf76e1c012" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 34667 } } peers { permanent_uuid: "9598e60340f742318ed8b5c447c653d5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43233 } }
I20260810 06:24:06.352108 29206 leader_election.cc:288] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 [CANDIDATE]: Term 1 election: Requested vote from peers aac54102787c4dd1aebed44d044bd9e5 (127.31.204.194:41499), c283d6f8d1f9460db1e8bbdf76e1c012 (127.31.204.195:34667)
I20260810 06:24:06.352422 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c0c2d300fe4409b9084eef514bc45d" candidate_uuid: "9598e60340f742318ed8b5c447c653d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aac54102787c4dd1aebed44d044bd9e5"
I20260810 06:24:06.352447 29093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c0c2d300fe4409b9084eef514bc45d" candidate_uuid: "9598e60340f742318ed8b5c447c653d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c283d6f8d1f9460db1e8bbdf76e1c012"
I20260810 06:24:06.352541 29093 raft_consensus.cc:3037] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:06.352540 28874 raft_consensus.cc:3037] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:06.353783 28874 raft_consensus.cc:2454] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9598e60340f742318ed8b5c447c653d5 in term 1.
I20260810 06:24:06.353783 29093 raft_consensus.cc:2454] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9598e60340f742318ed8b5c447c653d5 in term 1.
I20260810 06:24:06.354122 28543 leader_election.cc:302] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9598e60340f742318ed8b5c447c653d5, c283d6f8d1f9460db1e8bbdf76e1c012; no voters:
I20260810 06:24:06.354279 29206 raft_consensus.cc:2781] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:06.354449 29206 raft_consensus.cc:686] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 [term 1 LEADER]: Becoming Leader. State: Replica: 9598e60340f742318ed8b5c447c653d5, State: Running, Role: LEADER
W20260810 06:24:06.354705 28956 tablet.cc:2369] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:24:06.354763 29206 consensus_queue.cc:237] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac54102787c4dd1aebed44d044bd9e5" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41499 } } peers { permanent_uuid: "c283d6f8d1f9460db1e8bbdf76e1c012" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 34667 } } peers { permanent_uuid: "9598e60340f742318ed8b5c447c653d5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43233 } }
I20260810 06:24:06.357781 28411 catalog_manager.cc:5495] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9598e60340f742318ed8b5c447c653d5 (127.31.204.193). New cstate: current_term: 1 leader_uuid: "9598e60340f742318ed8b5c447c653d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac54102787c4dd1aebed44d044bd9e5" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c283d6f8d1f9460db1e8bbdf76e1c012" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 34667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9598e60340f742318ed8b5c447c653d5" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 43233 } health_report { overall_health: HEALTHY } } }
I20260810 06:24:06.393993 29193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:06.390621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36286 (local address 127.31.204.193:43233)
0810 06:24:06.390756 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:06.390761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:06.390775 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:24:06.390925 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:06.390928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:06.390982 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:06.391069 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:24:06.391323 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.391805 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.392644 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.392855 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.392941 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:06.393237 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:24:06.393629 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:24:06.393731 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:06.393813 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 06:24:06.393869 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 06:24:06.399380 29093 raft_consensus.cc:1261] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012 [term 1 FOLLOWER]: Refusing update from remote peer 9598e60340f742318ed8b5c447c653d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:24:06.399379 28874 raft_consensus.cc:1261] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5 [term 1 FOLLOWER]: Refusing update from remote peer 9598e60340f742318ed8b5c447c653d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:24:06.400023 29209 consensus_queue.cc:1035] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aac54102787c4dd1aebed44d044bd9e5" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:24:06.400192 29206 consensus_queue.cc:1035] T 16c0c2d300fe4409b9084eef514bc45d P 9598e60340f742318ed8b5c447c653d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c283d6f8d1f9460db1e8bbdf76e1c012" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 34667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:24:06.406271 29212 mvcc.cc:205] Tried to move back new op lower bound from 7316861118045388800 to 7316861117871751168. Current Snapshot: MvccSnapshot[applied={T|T < 7316861118045388800}]
I20260810 06:24:06.408341 29215 mvcc.cc:205] Tried to move back new op lower bound from 7316861118045388800 to 7316861117871751168. Current Snapshot: MvccSnapshot[applied={T|T < 7316861118045388800 or (T in {7316861118045388800})}]
I20260810 06:24:06.415997 28412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:24:06.416142 28412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:24:06.418478 28412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ab4ac108548481281cd608d693d6765: Sending DeleteTablet for 3 replicas of tablet 16c0c2d300fe4409b9084eef514bc45d
I20260810 06:24:06.419162 29059 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c0c2d300fe4409b9084eef514bc45d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:24:06 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:43064
I20260810 06:24:06.419183 28839 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c0c2d300fe4409b9084eef514bc45d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:24:06 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:59474
I20260810 06:24:06.419585 29228 tablet_replica.cc:330] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5: stopping tablet replica
I20260810 06:24:06.419582 29226 tablet_replica.cc:330] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012: stopping tablet replica
I20260810 06:24:06.419868 29226 raft_consensus.cc:2229] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:24:06.419891 29228 raft_consensus.cc:2229] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:24:06.419900 28608 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c0c2d300fe4409b9084eef514bc45d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:24:06 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:36280
I20260810 06:24:06.420171 29228 raft_consensus.cc:2258] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:24:06.420179 29226 raft_consensus.cc:2258] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:24:06.420316 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 28507
I20260810 06:24:06.423820 29226 ts_tablet_manager.cc:1905] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:24:06.426818 29226 ts_tablet_manager.cc:1918] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:24:06.426971 29226 log.cc:1196] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/16c0c2d300fe4409b9084eef514bc45d
I20260810 06:24:06.427386 29226 ts_tablet_manager.cc:1939] T 16c0c2d300fe4409b9084eef514bc45d P c283d6f8d1f9460db1e8bbdf76e1c012: Deleting consensus metadata
I20260810 06:24:06.428574 28394 catalog_manager.cc:4841] TS c283d6f8d1f9460db1e8bbdf76e1c012 (127.31.204.195:34667): tablet 16c0c2d300fe4409b9084eef514bc45d (table test-table [id=c121284c54fb41d09c8bdea344d71872]) successfully deleted
I20260810 06:24:06.429562 29228 ts_tablet_manager.cc:1905] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:24:06.431536 29228 ts_tablet_manager.cc:1918] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:24:06.431622 29228 log.cc:1196] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/16c0c2d300fe4409b9084eef514bc45d
I20260810 06:24:06.431892 29228 ts_tablet_manager.cc:1939] T 16c0c2d300fe4409b9084eef514bc45d P aac54102787c4dd1aebed44d044bd9e5: Deleting consensus metadata
I20260810 06:24:06.432811 28390 catalog_manager.cc:4841] TS aac54102787c4dd1aebed44d044bd9e5 (127.31.204.194:41499): tablet 16c0c2d300fe4409b9084eef514bc45d (table test-table [id=c121284c54fb41d09c8bdea344d71872]) successfully deleted
W20260810 06:24:06.433176 28390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.204.193:43233 (error 108)
I20260810 06:24:06.433440 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 28730
I20260810 06:24:06.435600 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:06.435330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:43233 (local address 127.0.0.1:36290)
0810 06:24:06.435473 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:06.435528 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:43233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260810 06:24:06.435729 28390 catalog_manager.cc:4584] TS 9598e60340f742318ed8b5c447c653d5 (127.31.204.193:43233): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 16c0c2d300fe4409b9084eef514bc45d: Network error: Client connection negotiation failed: client connection to 127.31.204.193:43233: connect: Connection refused (error 111)
I20260810 06:24:06.444207 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 28963
I20260810 06:24:06.457337 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 28353
2026-08-10T06:24:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2572 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T06:24:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:06Z Disabled control of system clock
I20260810 06:24:06.498164 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36601
--webserver_interface=127.31.204.254
--webserver_port=0
--builtin_ntp_servers=127.31.204.212:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:36601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 06:24:06.603793 29239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:06.604012 29239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:06.604048 29239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:06.607960 29239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:06.608008 29239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:06.608022 29239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:06.608052 29239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:06.608078 29239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:06.612597 29239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:36601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:06.613480 29239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:06.614734 29239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:06.622547 29247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:06.622624 29239 server_base.cc:1034] running on GCE node
W20260810 06:24:06.622713 29254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:06.622570 29249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:06.623358 29239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:06.624415 29239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:06.625587 29239 hybrid_clock.cc:648] HybridClock initialized: now 1786343046625564 us; error 34 us; skew 500 ppm
I20260810 06:24:06.627554 29239 webserver.cc:460] Webserver started at http://127.31.204.254:43707/ using document root <none> and password file <none>
I20260810 06:24:06.628100 29239 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:06.628167 29239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:06.628371 29239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:06.630540 29239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "07e2d826d5ac49419b5e41e5e75db4df"
format_stamp: "Formatted at 2026-08-10 06:24:06 on dist-test-slave-q9h9"
server_key: "66ed7f3376182e1de7b49cb648ef789d"
server_key_iv: "ee6d1ded0d805670cee60f288f0377cb"
server_key_version: "encryptionkey@0"
I20260810 06:24:06.631073 29239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "07e2d826d5ac49419b5e41e5e75db4df"
format_stamp: "Formatted at 2026-08-10 06:24:06 on dist-test-slave-q9h9"
server_key: "66ed7f3376182e1de7b49cb648ef789d"
server_key_iv: "ee6d1ded0d805670cee60f288f0377cb"
server_key_version: "encryptionkey@0"
I20260810 06:24:06.634578 29239 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:24:06.636878 29262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:06.638096 29239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:24:06.638201 29239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "07e2d826d5ac49419b5e41e5e75db4df"
format_stamp: "Formatted at 2026-08-10 06:24:06 on dist-test-slave-q9h9"
server_key: "66ed7f3376182e1de7b49cb648ef789d"
server_key_iv: "ee6d1ded0d805670cee60f288f0377cb"
server_key_version: "encryptionkey@0"
I20260810 06:24:06.638299 29239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:06.650360 29239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:06.650959 29239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:06.651134 29239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:06.657867 29239 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:36601
I20260810 06:24:06.657876 29349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:36601 every 8 connection(s)
I20260810 06:24:06.658771 29239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:06.661595 29351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:06.663285 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 29239
I20260810 06:24:06.663405 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:06.663684 32563 external_mini_cluster.cc:1428] Setting key 4cc755195c320437cd9eb69c62c552b7
I20260810 06:24:06.667598 29351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df: Bootstrap starting.
I20260810 06:24:06.670037 29351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:06.670931 29351 log.cc:824] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:06.672659 29351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df: No bootstrap required, opened a new log
I20260810 06:24:06.673373 29356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:06.664966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.31.204.254:36601)
0810 06:24:06.665606 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:06.665626 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:06.665669 (+ 43us) server_negotiation.cc:407] Connection header received
0810 06:24:06.666548 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:06.666569 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:06.667035 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:06.667399 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:06.667806 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.669201 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.670128 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.670784 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.670935 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:06.671720 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:06.671726 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:06.671737 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:06.672114 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:06.672173 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:06.672194 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:06.672290 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:06.672433 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 06:24:06.672687 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":105,"threads_started":1}
I20260810 06:24:06.675989 29351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e2d826d5ac49419b5e41e5e75db4df" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36601 } }
I20260810 06:24:06.676200 29351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:06.676265 29351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07e2d826d5ac49419b5e41e5e75db4df, State: Initialized, Role: FOLLOWER
I20260810 06:24:06.676628 29351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e2d826d5ac49419b5e41e5e75db4df" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36601 } }
I20260810 06:24:06.676744 29351 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:06.676805 29351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:06.676896 29351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:06.677685 29351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e2d826d5ac49419b5e41e5e75db4df" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36601 } }
I20260810 06:24:06.677975 29351 leader_election.cc:302] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07e2d826d5ac49419b5e41e5e75db4df; no voters:
I20260810 06:24:06.678313 29351 leader_election.cc:288] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:06.678398 29359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:06.678630 29359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df [term 1 LEADER]: Becoming Leader. State: Replica: 07e2d826d5ac49419b5e41e5e75db4df, State: Running, Role: LEADER
I20260810 06:24:06.678977 29359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e2d826d5ac49419b5e41e5e75db4df" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36601 } }
I20260810 06:24:06.679484 29351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:06.680606 29360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07e2d826d5ac49419b5e41e5e75db4df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e2d826d5ac49419b5e41e5e75db4df" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36601 } } }
I20260810 06:24:06.680606 29361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07e2d826d5ac49419b5e41e5e75db4df. Latest consensus state: current_term: 1 leader_uuid: "07e2d826d5ac49419b5e41e5e75db4df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e2d826d5ac49419b5e41e5e75db4df" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36601 } } }
I20260810 06:24:06.680763 29360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:06.680763 29361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:06.681857 29374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:06.684378 29374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:06.689097 29374 catalog_manager.cc:1347] Generated new cluster ID: 6279d5d73b9a441ba393a4bac6499cd9
I20260810 06:24:06.689210 29374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:06.703948 29374 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:06.704994 29374 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:06.714531 29374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df: Generated new TSK 0
I20260810 06:24:06.715174 29374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:06.717818 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36601
--builtin_ntp_servers=127.31.204.212:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 06:24:06.829345 29392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:06.829661 29392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:06.829698 29392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:06.833602 29392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:06.833652 29392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:06.833702 29392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:24:06.838236 29392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:06.839138 29392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:06.840408 29392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:06.849476 29404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:06.849514 29401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:06.849517 29402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:06.849722 29392 server_base.cc:1034] running on GCE node
I20260810 06:24:06.850160 29392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:06.850739 29392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:06.851889 29392 hybrid_clock.cc:648] HybridClock initialized: now 1786343046851875 us; error 25 us; skew 500 ppm
I20260810 06:24:06.853505 29392 webserver.cc:460] Webserver started at http://127.31.204.193:45557/ using document root <none> and password file <none>
I20260810 06:24:06.854045 29392 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:06.854112 29392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:06.854316 29392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:06.856204 29392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "1c146948d47d4bec9e04f499a6dfff4f"
format_stamp: "Formatted at 2026-08-10 06:24:06 on dist-test-slave-q9h9"
server_key: "487262c5353951b714e86504dab76ac6"
server_key_iv: "67b1b934c4c6a010af03a0382a404ce4"
server_key_version: "encryptionkey@0"
I20260810 06:24:06.856787 29392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "1c146948d47d4bec9e04f499a6dfff4f"
format_stamp: "Formatted at 2026-08-10 06:24:06 on dist-test-slave-q9h9"
server_key: "487262c5353951b714e86504dab76ac6"
server_key_iv: "67b1b934c4c6a010af03a0382a404ce4"
server_key_version: "encryptionkey@0"
I20260810 06:24:06.860181 29392 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:24:06.862358 29418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:06.863437 29392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:24:06.863538 29392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "1c146948d47d4bec9e04f499a6dfff4f"
format_stamp: "Formatted at 2026-08-10 06:24:06 on dist-test-slave-q9h9"
server_key: "487262c5353951b714e86504dab76ac6"
server_key_iv: "67b1b934c4c6a010af03a0382a404ce4"
server_key_version: "encryptionkey@0"
I20260810 06:24:06.863623 29392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:06.898476 29392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:06.899121 29392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:06.899299 29392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:06.899856 29392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:06.900763 29392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:06.900810 29392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:06.900863 29392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:06.900892 29392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:06.909322 29392 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:39727
I20260810 06:24:06.909354 29605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:39727 every 8 connection(s)
I20260810 06:24:06.910163 29392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:24:06.915051 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 29392
I20260810 06:24:06.915159 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:24:06.915421 32563 external_mini_cluster.cc:1428] Setting key 625848ef1f137b9d3ec24f2ef09d40ec
I20260810 06:24:06.917263 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36601
--builtin_ntp_servers=127.31.204.212:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 06:24:06.919238 29356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:06.911827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:59911 (local address 127.31.204.254:36601)
0810 06:24:06.911954 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:06.911958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:06.912938 (+ 980us) server_negotiation.cc:407] Connection header received
0810 06:24:06.914306 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:06.914310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:06.914365 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:06.914435 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:06.915311 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.915930 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.917288 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.917573 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.917660 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:06.918751 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:06.918755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:06.918758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:06.918771 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:06.918793 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:06.918803 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:06.918868 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:06.919049 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 06:24:06.919101 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:24:06.920437 29608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:06.912188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36601 (local address 127.31.204.193:59911)
0810 06:24:06.912797 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:06.912834 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:06.914091 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:06.914614 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:06.914621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:06.914699 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:06.915174 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:06.915191 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.916051 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.916054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:06.917165 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:06.917175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:06.917709 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:06.917713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:06.917878 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:06.918468 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:24:06.918470 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:06.918521 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:24:06.918596 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:06.918908 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:06.918918 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:06.918924 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 06:24:06.919458 (+ 534us) client_negotiation.cc:240] Negotiation successful
0810 06:24:06.919669 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":108,"threads_started":1}
I20260810 06:24:06.921497 29606 heartbeater.cc:347] Connected to a master server at 127.31.204.254:36601
I20260810 06:24:06.921702 29606 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:06.922233 29606 heartbeater.cc:511] Master 127.31.204.254:36601 requested a full tablet report, sending...
I20260810 06:24:06.923594 29293 ts_manager.cc:194] Registered new tserver with Master: 1c146948d47d4bec9e04f499a6dfff4f (127.31.204.193:39727)
I20260810 06:24:06.924675 29293 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.204.193:59911
W20260810 06:24:07.024577 29610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:07.024804 29610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:07.024842 29610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:07.028766 29610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:07.028810 29610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:07.028869 29610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:24:07.033406 29610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:07.034332 29610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:07.035601 29610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:07.044412 29622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:07.044420 29619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:07.044552 29625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:07.044754 29610 server_base.cc:1034] running on GCE node
I20260810 06:24:07.045116 29610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:07.045689 29610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:07.046856 29610 hybrid_clock.cc:648] HybridClock initialized: now 1786343047046824 us; error 59 us; skew 500 ppm
I20260810 06:24:07.048485 29610 webserver.cc:460] Webserver started at http://127.31.204.194:36131/ using document root <none> and password file <none>
I20260810 06:24:07.049003 29610 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:07.049067 29610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:07.049299 29610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:07.051251 29610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "87c90ec723c648d2b40604caf77955e0"
format_stamp: "Formatted at 2026-08-10 06:24:07 on dist-test-slave-q9h9"
server_key: "4033dc5e9e597ba0cb9102374db98400"
server_key_iv: "69401098755b5b5a33921b497164d384"
server_key_version: "encryptionkey@0"
I20260810 06:24:07.051764 29610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "87c90ec723c648d2b40604caf77955e0"
format_stamp: "Formatted at 2026-08-10 06:24:07 on dist-test-slave-q9h9"
server_key: "4033dc5e9e597ba0cb9102374db98400"
server_key_iv: "69401098755b5b5a33921b497164d384"
server_key_version: "encryptionkey@0"
I20260810 06:24:07.055140 29610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:24:07.057288 29637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:07.058382 29610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:24:07.058483 29610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "87c90ec723c648d2b40604caf77955e0"
format_stamp: "Formatted at 2026-08-10 06:24:07 on dist-test-slave-q9h9"
server_key: "4033dc5e9e597ba0cb9102374db98400"
server_key_iv: "69401098755b5b5a33921b497164d384"
server_key_version: "encryptionkey@0"
I20260810 06:24:07.058564 29610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:07.081458 29610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:07.082094 29610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:07.082257 29610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:07.082791 29610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:07.083662 29610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:07.083707 29610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:07.083767 29610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:07.083794 29610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:07.091671 29610 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:37799
I20260810 06:24:07.091708 29826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:37799 every 8 connection(s)
I20260810 06:24:07.092492 29610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:24:07.093876 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 29610
I20260810 06:24:07.093986 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:24:07.094314 32563 external_mini_cluster.cc:1428] Setting key 6a19f674b473518ae1bb281d6793ae2a
I20260810 06:24:07.096266 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36601
--builtin_ntp_servers=127.31.204.212:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 06:24:07.101521 29356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:07.094141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:48003 (local address 127.31.204.254:36601)
0810 06:24:07.094296 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:07.094300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:07.095408 (+ 1108us) server_negotiation.cc:407] Connection header received
0810 06:24:07.096848 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:07.096851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:07.096902 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:07.096979 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:07.097771 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.098420 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.099809 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.100018 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.100104 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:07.101080 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:07.101083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:07.101085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:07.101096 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:07.101114 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:07.101121 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:07.101179 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:07.101323 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 06:24:07.101380 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 06:24:07.102320 29840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:07.094558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36601 (local address 127.31.204.194:48003)
0810 06:24:07.095257 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:07.095309 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:07.096652 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:07.097118 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:07.097124 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:07.097206 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:07.097646 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:07.097658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.098537 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.098541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:07.099686 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:07.099695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.100150 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.100154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:07.100310 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:07.100834 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:24:07.100836 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:07.100874 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:24:07.100933 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:07.101188 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:07.101197 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:07.101204 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 06:24:07.101307 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 06:24:07.101558 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":115,"threads_started":1}
I20260810 06:24:07.103461 29827 heartbeater.cc:347] Connected to a master server at 127.31.204.254:36601
I20260810 06:24:07.103659 29827 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:07.104111 29827 heartbeater.cc:511] Master 127.31.204.254:36601 requested a full tablet report, sending...
I20260810 06:24:07.105160 29293 ts_manager.cc:194] Registered new tserver with Master: 87c90ec723c648d2b40604caf77955e0 (127.31.204.194:37799)
I20260810 06:24:07.105916 29293 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.204.194:48003
W20260810 06:24:07.206678 29842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:07.206909 29842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:07.206944 29842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:07.210747 29842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:07.210789 29842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:07.210839 29842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:24:07.215379 29842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:36601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:07.216289 29842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:07.217545 29842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:07.226393 29858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:07.226388 29854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:07.226385 29856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:07.226519 29842 server_base.cc:1034] running on GCE node
I20260810 06:24:07.226979 29842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:07.227531 29842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:07.228672 29842 hybrid_clock.cc:648] HybridClock initialized: now 1786343047228648 us; error 39 us; skew 500 ppm
I20260810 06:24:07.230357 29842 webserver.cc:460] Webserver started at http://127.31.204.195:45935/ using document root <none> and password file <none>
I20260810 06:24:07.230858 29842 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:07.230926 29842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:07.231284 29842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:07.233363 29842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "9bc735d9a75b4d6d9e1936de636cc77a"
format_stamp: "Formatted at 2026-08-10 06:24:07 on dist-test-slave-q9h9"
server_key: "a6aed6c8192a06159956436c587cc604"
server_key_iv: "4bfad822e360c86987f8de75dd009d4a"
server_key_version: "encryptionkey@0"
I20260810 06:24:07.233933 29842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "9bc735d9a75b4d6d9e1936de636cc77a"
format_stamp: "Formatted at 2026-08-10 06:24:07 on dist-test-slave-q9h9"
server_key: "a6aed6c8192a06159956436c587cc604"
server_key_iv: "4bfad822e360c86987f8de75dd009d4a"
server_key_version: "encryptionkey@0"
I20260810 06:24:07.237512 29842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:24:07.239879 29867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:07.241006 29842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:07.241114 29842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "9bc735d9a75b4d6d9e1936de636cc77a"
format_stamp: "Formatted at 2026-08-10 06:24:07 on dist-test-slave-q9h9"
server_key: "a6aed6c8192a06159956436c587cc604"
server_key_iv: "4bfad822e360c86987f8de75dd009d4a"
server_key_version: "encryptionkey@0"
I20260810 06:24:07.241209 29842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:07.264089 29842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:07.264709 29842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:07.264861 29842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:07.265411 29842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:07.266348 29842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:07.266395 29842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:07.266433 29842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:07.266453 29842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:07.274730 29842 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:42317
I20260810 06:24:07.274789 30070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:42317 every 8 connection(s)
I20260810 06:24:07.275580 29842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:24:07.283208 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 29842
I20260810 06:24:07.283311 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:24:07.283557 32563 external_mini_cluster.cc:1428] Setting key 8c84fce233002c3fb37c69467256ec2e
I20260810 06:24:07.284662 29356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:07.277333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:45129 (local address 127.31.204.254:36601)
0810 06:24:07.277462 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:07.277466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:07.278500 (+ 1034us) server_negotiation.cc:407] Connection header received
0810 06:24:07.279973 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:07.279977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:07.280042 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:07.280131 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:07.281029 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.281587 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.282784 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.283018 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.283116 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:07.284215 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:07.284218 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:07.284220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:07.284232 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:07.284252 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:07.284260 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:07.284320 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:07.284458 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 06:24:07.284520 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 06:24:07.285470 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:07.277656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36601 (local address 127.31.204.195:45129)
0810 06:24:07.278356 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:07.278397 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:07.279737 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:07.280305 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:07.280312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:07.280395 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:07.280881 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:07.280895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.281724 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.281727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:07.282643 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:07.282654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.283167 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.283172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:07.283351 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:07.283963 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:24:07.283965 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:07.284004 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:24:07.284065 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:07.284330 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:07.284339 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:07.284345 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 06:24:07.284450 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 06:24:07.284711 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":115,"threads_started":1}
I20260810 06:24:07.286520 30071 heartbeater.cc:347] Connected to a master server at 127.31.204.254:36601
I20260810 06:24:07.286744 30071 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:07.287257 30071 heartbeater.cc:511] Master 127.31.204.254:36601 requested a full tablet report, sending...
I20260810 06:24:07.288194 29293 ts_manager.cc:194] Registered new tserver with Master: 9bc735d9a75b4d6d9e1936de636cc77a (127.31.204.195:42317)
I20260810 06:24:07.288754 29293 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.204.195:45129
I20260810 06:24:07.297103 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:24:07.302109 29356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:07.299006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.31.204.254:36601)
0810 06:24:07.299106 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:07.299109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:07.299293 (+ 184us) server_negotiation.cc:407] Connection header received
0810 06:24:07.299385 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:07.299387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:07.299422 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:07.299479 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:07.299666 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.300143 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.300889 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.301081 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.301143 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:07.301801 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:07.301804 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:07.301806 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:07.301814 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:07.301827 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:07.301833 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:07.301873 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:07.301930 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 06:24:07.301970 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 06:24:07.304520 29293 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:24:07.307405 29293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:24:07.320427 30090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:07.315067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:37799 (local address 127.0.0.1:41672)
0810 06:24:07.315919 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:07.315932 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:07.316013 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:07.316568 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:07.316574 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:07.316595 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:07.316784 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:07.316790 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.318102 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.318105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:07.319194 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:07.319202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.319927 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.319931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:07.320034 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:07.320068 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 06:24:07.320169 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 06:24:07.320290 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":42,"spinlock_wait_cycles":4992,"thread_start_us":86,"threads_started":1}
I20260810 06:24:07.320679 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:07.315712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.31.204.194:37799)
0810 06:24:07.316110 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:07.316122 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:07.316146 (+ 24us) server_negotiation.cc:407] Connection header received
0810 06:24:07.316249 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:07.316255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:07.316434 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:07.316564 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:07.316892 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.317969 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.319312 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.319821 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.319929 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:07.320100 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:07.320416 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 06:24:07.320520 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":90,"threads_started":1}
I20260810 06:24:07.320737 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:07.315091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.31.204.195:42317)
0810 06:24:07.316267 (+ 1176us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:07.316279 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:07.316299 (+ 20us) server_negotiation.cc:407] Connection header received
0810 06:24:07.316353 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:07.316356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:07.316487 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:07.316610 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:07.316923 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.317737 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.319285 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.320041 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.320151 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:07.320314 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:07.320494 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 06:24:07.320590 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1075,"thread_start_us":104,"threads_started":1}
I20260810 06:24:07.320966 30091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:07.314962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:42317 (local address 127.0.0.1:56474)
0810 06:24:07.315428 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:07.315477 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:07.315594 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:07.316608 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:07.316611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:07.316629 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:07.316818 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:07.316824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.317834 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.317837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:07.319156 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:07.319167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.320178 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.320182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:07.320273 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:07.320294 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 06:24:07.320372 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 06:24:07.320424 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":39,"thread_start_us":63,"threads_started":1}
I20260810 06:24:07.321141 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:07.315637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:39727 (local address 127.0.0.1:38874)
0810 06:24:07.316181 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:07.316192 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:07.316265 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:07.317449 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:07.317452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:07.317471 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:07.317545 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:07.317550 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.318773 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.318776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:07.319753 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:07.319764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.320794 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.320798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:07.320883 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:07.320908 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 06:24:07.320959 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 06:24:07.321022 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":86,"threads_started":1}
I20260810 06:24:07.321933 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:07.315873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38874 (local address 127.31.204.193:39727)
0810 06:24:07.317048 (+ 1175us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:07.317062 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:07.317084 (+ 22us) server_negotiation.cc:407] Connection header received
0810 06:24:07.317151 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:07.317156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:07.317316 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:07.317447 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:07.317645 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.318636 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.319909 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.320636 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.321483 (+ 847us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:07.321638 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:07.321685 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 06:24:07.321783 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1069,"thread_start_us":112,"threads_started":1}
I20260810 06:24:07.323650 29949 tablet_service.cc:1467] Processing CreateTablet for tablet 7de0765999c147ccbeefb9093a0fe828 (DEFAULT_TABLE table=test-table [id=7af4c0ea28e24d04850f2ecff3d7f697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:24:07.323650 29715 tablet_service.cc:1467] Processing CreateTablet for tablet 7de0765999c147ccbeefb9093a0fe828 (DEFAULT_TABLE table=test-table [id=7af4c0ea28e24d04850f2ecff3d7f697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:24:07.324244 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 7de0765999c147ccbeefb9093a0fe828 (DEFAULT_TABLE table=test-table [id=7af4c0ea28e24d04850f2ecff3d7f697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:24:07.324733 29949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de0765999c147ccbeefb9093a0fe828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:07.324733 29715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de0765999c147ccbeefb9093a0fe828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:07.325067 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de0765999c147ccbeefb9093a0fe828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:07.331333 30099 tablet_bootstrap.cc:492] T 7de0765999c147ccbeefb9093a0fe828 P 1c146948d47d4bec9e04f499a6dfff4f: Bootstrap starting.
I20260810 06:24:07.331741 30100 tablet_bootstrap.cc:492] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0: Bootstrap starting.
I20260810 06:24:07.332139 30102 tablet_bootstrap.cc:492] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a: Bootstrap starting.
I20260810 06:24:07.333617 30099 tablet_bootstrap.cc:654] T 7de0765999c147ccbeefb9093a0fe828 P 1c146948d47d4bec9e04f499a6dfff4f: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:07.334285 30102 tablet_bootstrap.cc:654] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:07.334506 30099 log.cc:824] T 7de0765999c147ccbeefb9093a0fe828 P 1c146948d47d4bec9e04f499a6dfff4f: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:07.335088 30102 log.cc:824] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:07.335530 30100 tablet_bootstrap.cc:654] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:07.336310 30100 log.cc:824] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:07.336899 30102 tablet_bootstrap.cc:492] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a: No bootstrap required, opened a new log
I20260810 06:24:07.336937 30099 tablet_bootstrap.cc:492] T 7de0765999c147ccbeefb9093a0fe828 P 1c146948d47d4bec9e04f499a6dfff4f: No bootstrap required, opened a new log
I20260810 06:24:07.337069 30102 ts_tablet_manager.cc:1397] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:24:07.337112 30099 ts_tablet_manager.cc:1397] T 7de0765999c147ccbeefb9093a0fe828 P 1c146948d47d4bec9e04f499a6dfff4f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 06:24:07.338200 30100 tablet_bootstrap.cc:492] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0: No bootstrap required, opened a new log
I20260810 06:24:07.338370 30100 ts_tablet_manager.cc:1397] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 06:24:07.340093 30102 raft_consensus.cc:348] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc735d9a75b4d6d9e1936de636cc77a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42317 } } peers { permanent_uuid: "87c90ec723c648d2b40604caf77955e0" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37799 } } peers { permanent_uuid: "1c146948d47d4bec9e04f499a6dfff4f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39727 } }
I20260810 06:24:07.340296 30102 raft_consensus.cc:374] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:07.340365 30102 raft_consensus.cc:729] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc735d9a75b4d6d9e1936de636cc77a, State: Initialized, Role: FOLLOWER
I20260810 06:24:07.340758 30102 consensus_queue.cc:260] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc735d9a75b4d6d9e1936de636cc77a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42317 } } peers { permanent_uuid: "87c90ec723c648d2b40604caf77955e0" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37799 } } peers { permanent_uuid: "1c146948d47d4bec9e04f499a6dfff4f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39727 } }
I20260810 06:24:07.341364 30071 heartbeater.cc:503] Master 127.31.204.254:36601 was elected leader, sending a full tablet report...
I20260810 06:24:07.341224 30099 raft_consensus.cc:348] T 7de0765999c147ccbeefb9093a0fe828 P 1c146948d47d4bec9e04f499a6dfff4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc735d9a75b4d6d9e1936de636cc77a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42317 } } peers { permanent_uuid: "87c90ec723c648d2b40604caf77955e0" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37799 } } peers { permanent_uuid: "1c146948d47d4bec9e04f499a6dfff4f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39727 } }
I20260810 06:24:07.341454 30099 raft_consensus.cc:374] T 7de0765999c147ccbeefb9093a0fe828 P 1c146948d47d4bec9e04f499a6dfff4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:07.341501 30099 raft_consensus.cc:729] T 7de0765999c147ccbeefb9093a0fe828 P 1c146948d47d4bec9e04f499a6dfff4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c146948d47d4bec9e04f499a6dfff4f, State: Initialized, Role: FOLLOWER
I20260810 06:24:07.341722 30102 ts_tablet_manager.cc:1428] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 06:24:07.341933 30099 consensus_queue.cc:260] T 7de0765999c147ccbeefb9093a0fe828 P 1c146948d47d4bec9e04f499a6dfff4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc735d9a75b4d6d9e1936de636cc77a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42317 } } peers { permanent_uuid: "87c90ec723c648d2b40604caf77955e0" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37799 } } peers { permanent_uuid: "1c146948d47d4bec9e04f499a6dfff4f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39727 } }
I20260810 06:24:07.342447 30100 raft_consensus.cc:348] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc735d9a75b4d6d9e1936de636cc77a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42317 } } peers { permanent_uuid: "87c90ec723c648d2b40604caf77955e0" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37799 } } peers { permanent_uuid: "1c146948d47d4bec9e04f499a6dfff4f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39727 } }
I20260810 06:24:07.342970 30100 raft_consensus.cc:374] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:07.343048 30099 ts_tablet_manager.cc:1428] T 7de0765999c147ccbeefb9093a0fe828 P 1c146948d47d4bec9e04f499a6dfff4f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 06:24:07.343072 30100 raft_consensus.cc:729] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c90ec723c648d2b40604caf77955e0, State: Initialized, Role: FOLLOWER
I20260810 06:24:07.343456 29606 heartbeater.cc:503] Master 127.31.204.254:36601 was elected leader, sending a full tablet report...
I20260810 06:24:07.343547 30100 consensus_queue.cc:260] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc735d9a75b4d6d9e1936de636cc77a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42317 } } peers { permanent_uuid: "87c90ec723c648d2b40604caf77955e0" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37799 } } peers { permanent_uuid: "1c146948d47d4bec9e04f499a6dfff4f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39727 } }
I20260810 06:24:07.344113 29827 heartbeater.cc:503] Master 127.31.204.254:36601 was elected leader, sending a full tablet report...
I20260810 06:24:07.344420 30100 ts_tablet_manager.cc:1428] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260810 06:24:07.411270 29607 tablet.cc:2369] T 7de0765999c147ccbeefb9093a0fe828 P 1c146948d47d4bec9e04f499a6dfff4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:24:07.469313 30107 raft_consensus.cc:482] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:24:07.469527 30107 raft_consensus.cc:504] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc735d9a75b4d6d9e1936de636cc77a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42317 } } peers { permanent_uuid: "87c90ec723c648d2b40604caf77955e0" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37799 } } peers { permanent_uuid: "1c146948d47d4bec9e04f499a6dfff4f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39727 } }
I20260810 06:24:07.470662 30107 leader_election.cc:288] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87c90ec723c648d2b40604caf77955e0 (127.31.204.194:37799), 1c146948d47d4bec9e04f499a6dfff4f (127.31.204.193:39727)
I20260810 06:24:07.475222 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:07.470868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:37799 (local address 127.31.204.195:59009)
0810 06:24:07.471075 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:07.471089 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:07.471199 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:07.471449 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:07.471451 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:07.471467 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:07.471571 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:07.471577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.472361 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.472365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:07.473560 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:07.473572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.474644 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.474647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:07.474741 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:07.474876 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 06:24:07.475010 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 06:24:07.475093 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":68}
I20260810 06:24:07.475222 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:07.470996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:44103 (local address 127.31.204.193:39727)
0810 06:24:07.471103 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:07.471107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:07.471375 (+ 268us) server_negotiation.cc:407] Connection header received
0810 06:24:07.471491 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:07.471494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:07.471538 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:07.471599 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:07.471852 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.472463 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.473801 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.474435 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.474551 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:07.474599 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:07.475014 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 06:24:07.475093 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:24:07.475246 30112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:07.470900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:39727 (local address 127.31.204.195:44103)
0810 06:24:07.471284 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:07.471295 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:07.471396 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:07.471618 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:07.471621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:07.471636 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:07.471749 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:07.471754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.472592 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.472613 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:07.473668 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:07.473679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.474584 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.474587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:07.474837 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:07.474880 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 06:24:07.474995 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 06:24:07.475081 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":80,"thread_start_us":129,"threads_started":1}
I20260810 06:24:07.475793 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:07.470966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:59009 (local address 127.31.204.194:37799)
0810 06:24:07.471087 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:07.471091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:07.471147 (+ 56us) server_negotiation.cc:407] Connection header received
0810 06:24:07.471315 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:07.471318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:07.471365 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:07.471428 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:07.471681 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.472228 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.473712 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.474405 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.474525 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:07.474573 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:07.475631 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0810 06:24:07.475696 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:24:07.475831 29526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de0765999c147ccbeefb9093a0fe828" candidate_uuid: "9bc735d9a75b4d6d9e1936de636cc77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c146948d47d4bec9e04f499a6dfff4f" is_pre_election: true
I20260810 06:24:07.476126 29526 raft_consensus.cc:2454] T 7de0765999c147ccbeefb9093a0fe828 P 1c146948d47d4bec9e04f499a6dfff4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc735d9a75b4d6d9e1936de636cc77a in term 0.
I20260810 06:24:07.476286 29748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de0765999c147ccbeefb9093a0fe828" candidate_uuid: "9bc735d9a75b4d6d9e1936de636cc77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c90ec723c648d2b40604caf77955e0" is_pre_election: true
I20260810 06:24:07.476559 29748 raft_consensus.cc:2454] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc735d9a75b4d6d9e1936de636cc77a in term 0.
I20260810 06:24:07.476570 29869 leader_election.cc:302] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c146948d47d4bec9e04f499a6dfff4f, 9bc735d9a75b4d6d9e1936de636cc77a; no voters:
I20260810 06:24:07.476817 30107 raft_consensus.cc:2781] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:24:07.476898 30107 raft_consensus.cc:482] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:24:07.476941 30107 raft_consensus.cc:3037] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:07.477931 30107 raft_consensus.cc:504] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc735d9a75b4d6d9e1936de636cc77a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42317 } } peers { permanent_uuid: "87c90ec723c648d2b40604caf77955e0" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37799 } } peers { permanent_uuid: "1c146948d47d4bec9e04f499a6dfff4f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39727 } }
I20260810 06:24:07.478273 30107 leader_election.cc:288] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [CANDIDATE]: Term 1 election: Requested vote from peers 87c90ec723c648d2b40604caf77955e0 (127.31.204.194:37799), 1c146948d47d4bec9e04f499a6dfff4f (127.31.204.193:39727)
I20260810 06:24:07.478572 29748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de0765999c147ccbeefb9093a0fe828" candidate_uuid: "9bc735d9a75b4d6d9e1936de636cc77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c90ec723c648d2b40604caf77955e0"
I20260810 06:24:07.478598 29526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de0765999c147ccbeefb9093a0fe828" candidate_uuid: "9bc735d9a75b4d6d9e1936de636cc77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c146948d47d4bec9e04f499a6dfff4f"
I20260810 06:24:07.478675 29748 raft_consensus.cc:3037] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:07.478708 29526 raft_consensus.cc:3037] T 7de0765999c147ccbeefb9093a0fe828 P 1c146948d47d4bec9e04f499a6dfff4f [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:07.479713 29748 raft_consensus.cc:2454] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc735d9a75b4d6d9e1936de636cc77a in term 1.
I20260810 06:24:07.479737 29526 raft_consensus.cc:2454] T 7de0765999c147ccbeefb9093a0fe828 P 1c146948d47d4bec9e04f499a6dfff4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc735d9a75b4d6d9e1936de636cc77a in term 1.
I20260810 06:24:07.479995 29868 leader_election.cc:302] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87c90ec723c648d2b40604caf77955e0, 9bc735d9a75b4d6d9e1936de636cc77a; no voters:
I20260810 06:24:07.480155 30107 raft_consensus.cc:2781] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:07.480396 30107 raft_consensus.cc:686] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [term 1 LEADER]: Becoming Leader. State: Replica: 9bc735d9a75b4d6d9e1936de636cc77a, State: Running, Role: LEADER
I20260810 06:24:07.480705 30107 consensus_queue.cc:237] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc735d9a75b4d6d9e1936de636cc77a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42317 } } peers { permanent_uuid: "87c90ec723c648d2b40604caf77955e0" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37799 } } peers { permanent_uuid: "1c146948d47d4bec9e04f499a6dfff4f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39727 } }
I20260810 06:24:07.483865 29292 catalog_manager.cc:5495] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bc735d9a75b4d6d9e1936de636cc77a (127.31.204.195). New cstate: current_term: 1 leader_uuid: "9bc735d9a75b4d6d9e1936de636cc77a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc735d9a75b4d6d9e1936de636cc77a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 42317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87c90ec723c648d2b40604caf77955e0" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c146948d47d4bec9e04f499a6dfff4f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39727 } health_report { overall_health: UNKNOWN } } }
I20260810 06:24:07.504953 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:07.502063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56476 (local address 127.31.204.195:42317)
0810 06:24:07.502176 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:07.502180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:07.502206 (+ 26us) server_negotiation.cc:407] Connection header received
0810 06:24:07.502273 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:07.502275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:07.502324 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:07.502404 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:24:07.502581 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.503056 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.503814 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:07.504012 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:07.504079 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:07.504286 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:24:07.504668 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:24:07.504763 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:07.504807 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 06:24:07.504850 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:24:07.510337 29748 raft_consensus.cc:1261] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0 [term 1 FOLLOWER]: Refusing update from remote peer 9bc735d9a75b4d6d9e1936de636cc77a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:24:07.510337 29526 raft_consensus.cc:1261] T 7de0765999c147ccbeefb9093a0fe828 P 1c146948d47d4bec9e04f499a6dfff4f [term 1 FOLLOWER]: Refusing update from remote peer 9bc735d9a75b4d6d9e1936de636cc77a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:24:07.510964 30113 consensus_queue.cc:1035] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c146948d47d4bec9e04f499a6dfff4f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 39727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:24:07.511135 30107 consensus_queue.cc:1035] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [LEADER]: Connected to new peer: Peer: permanent_uuid: "87c90ec723c648d2b40604caf77955e0" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:24:07.518497 30122 mvcc.cc:205] Tried to move back new op lower bound from 7316861122595733504 to 7316861122483380224. Current Snapshot: MvccSnapshot[applied={T|T < 7316861122595733504}]
I20260810 06:24:07.521552 30123 mvcc.cc:205] Tried to move back new op lower bound from 7316861122595733504 to 7316861122483380224. Current Snapshot: MvccSnapshot[applied={T|T < 7316861122595733504}]
I20260810 06:24:07.526386 29293 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:24:07.526551 29293 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:24:07.528839 29293 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07e2d826d5ac49419b5e41e5e75db4df: Sending DeleteTablet for 3 replicas of tablet 7de0765999c147ccbeefb9093a0fe828
I20260810 06:24:07.529574 29715 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de0765999c147ccbeefb9093a0fe828 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:24:07 UTC) from {username='slave'} at 127.0.0.1:41672
I20260810 06:24:07.529651 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de0765999c147ccbeefb9093a0fe828 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:24:07 UTC) from {username='slave'} at 127.0.0.1:38874
I20260810 06:24:07.530059 30139 tablet_replica.cc:330] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0: stopping tablet replica
I20260810 06:24:07.530061 29949 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de0765999c147ccbeefb9093a0fe828 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:24:07 UTC) from {username='slave'} at 127.0.0.1:56474
I20260810 06:24:07.530213 30143 tablet_replica.cc:330] T 7de0765999c147ccbeefb9093a0fe828 P 1c146948d47d4bec9e04f499a6dfff4f: stopping tablet replica
I20260810 06:24:07.530470 30139 raft_consensus.cc:2229] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:24:07.530555 30143 raft_consensus.cc:2229] T 7de0765999c147ccbeefb9093a0fe828 P 1c146948d47d4bec9e04f499a6dfff4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:24:07.530776 30139 raft_consensus.cc:2258] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:24:07.530778 30143 raft_consensus.cc:2258] T 7de0765999c147ccbeefb9093a0fe828 P 1c146948d47d4bec9e04f499a6dfff4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:24:07.531299 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 29392
I20260810 06:24:07.534904 30139 ts_tablet_manager.cc:1905] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:24:07.537833 30139 ts_tablet_manager.cc:1918] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:24:07.537977 30139 log.cc:1196] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/7de0765999c147ccbeefb9093a0fe828
I20260810 06:24:07.538309 30139 ts_tablet_manager.cc:1939] T 7de0765999c147ccbeefb9093a0fe828 P 87c90ec723c648d2b40604caf77955e0: Deleting consensus metadata
I20260810 06:24:07.539440 29263 catalog_manager.cc:4841] TS 87c90ec723c648d2b40604caf77955e0 (127.31.204.194:37799): tablet 7de0765999c147ccbeefb9093a0fe828 (table test-table [id=7af4c0ea28e24d04850f2ecff3d7f697]) successfully deleted
I20260810 06:24:07.542229 30144 tablet_replica.cc:330] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a: stopping tablet replica
I20260810 06:24:07.542548 30144 raft_consensus.cc:2229] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:24:07.542970 30144 raft_consensus.cc:2258] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:24:07.544481 30144 ts_tablet_manager.cc:1905] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:24:07.547134 30144 ts_tablet_manager.cc:1918] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:24:07.547248 30144 log.cc:1196] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/7de0765999c147ccbeefb9093a0fe828
W20260810 06:24:07.547430 29265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.204.193:39727 (error 108)
I20260810 06:24:07.547530 30144 ts_tablet_manager.cc:1939] T 7de0765999c147ccbeefb9093a0fe828 P 9bc735d9a75b4d6d9e1936de636cc77a: Deleting consensus metadata
I20260810 06:24:07.548372 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:07.548106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:39727 (local address 127.0.0.1:38884)
0810 06:24:07.548221 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:07.548289 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 06:24:07.548492 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 29610
W20260810 06:24:07.555960 29265 catalog_manager.cc:4584] TS 1c146948d47d4bec9e04f499a6dfff4f (127.31.204.193:39727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7de0765999c147ccbeefb9093a0fe828: Network error: Client connection negotiation failed: client connection to 127.31.204.193:39727: connect: Connection refused (error 111)
I20260810 06:24:07.556176 29265 catalog_manager.cc:4841] TS 9bc735d9a75b4d6d9e1936de636cc77a (127.31.204.195:42317): tablet 7de0765999c147ccbeefb9093a0fe828 (table test-table [id=7af4c0ea28e24d04850f2ecff3d7f697]) successfully deleted
I20260810 06:24:07.564488 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 29842
I20260810 06:24:07.575626 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 29239
2026-08-10T06:24:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1112 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:07 dist-test-slave-q9h9 krb5kdc[30149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:07 dist-test-slave-q9h9 krb5kdc[30149](info): set up 2 sockets
Aug 10 06:24:07 dist-test-slave-q9h9 krb5kdc[30149](info): commencing operation
krb5kdc: starting...
W20260810 06:24:08.727811 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:08 dist-test-slave-q9h9 krb5kdc[30149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:24:08.837455 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:34165
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:34165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:08.942358 30168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:08.942579 30168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:08.942631 30168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:08.946486 30168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:08.946532 30168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:08.946558 30168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:08.946579 30168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:08.946607 30168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:08.951524 30168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:34165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:34165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:08.952409 30168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:08.953313 30168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:08.961151 30177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:08.961181 30179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:08.961180 30187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:08.961248 30168 server_base.cc:1034] running on GCE node
I20260810 06:24:08.961884 30168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:08.962899 30168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:08.964048 30168 hybrid_clock.cc:648] HybridClock initialized: now 1786343048964022 us; error 43 us; skew 500 ppm
Aug 10 06:24:08 dist-test-slave-q9h9 krb5kdc[30149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343048, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:08.969367 30168 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:24:08.970223 30168 webserver.cc:460] Webserver started at http://127.31.204.254:42729/ using document root <none> and password file <none>
I20260810 06:24:08.970721 30168 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:08.970788 30168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:08.970973 30168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:08.972852 30168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "53c3cd7d8905451f92daaac3927f1456"
format_stamp: "Formatted at 2026-08-10 06:24:08 on dist-test-slave-q9h9"
server_key: "6ec4068237e563c55d9e738182f9bcc2"
server_key_iv: "d2a78a5e71e4d6dbf237745e06f1cde9"
server_key_version: "encryptionkey@0"
I20260810 06:24:08.973369 30168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "53c3cd7d8905451f92daaac3927f1456"
format_stamp: "Formatted at 2026-08-10 06:24:08 on dist-test-slave-q9h9"
server_key: "6ec4068237e563c55d9e738182f9bcc2"
server_key_iv: "d2a78a5e71e4d6dbf237745e06f1cde9"
server_key_version: "encryptionkey@0"
I20260810 06:24:08.976907 30168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260810 06:24:08.979151 30198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:08.980291 30168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:24:08.980407 30168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "53c3cd7d8905451f92daaac3927f1456"
format_stamp: "Formatted at 2026-08-10 06:24:08 on dist-test-slave-q9h9"
server_key: "6ec4068237e563c55d9e738182f9bcc2"
server_key_iv: "d2a78a5e71e4d6dbf237745e06f1cde9"
server_key_version: "encryptionkey@0"
I20260810 06:24:08.980499 30168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:09.008678 30168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:09.011528 30168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:09.011701 30168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:09.018225 30168 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:34165
I20260810 06:24:09.018252 30287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:34165 every 8 connection(s)
I20260810 06:24:09.019074 30168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:09.021576 30288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:09.022470 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 30168
I20260810 06:24:09.022594 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:09.022848 32563 external_mini_cluster.cc:1428] Setting key 44ee2ca81dcf49ef77b459aba8d396e8
I20260810 06:24:09.027184 30288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456: Bootstrap starting.
I20260810 06:24:09.029264 30288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:09.029913 30288 log.cc:824] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456: Log is configured to *not* fsync() on all Append() calls
Aug 10 06:24:09 dist-test-slave-q9h9 krb5kdc[30149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:24:09.031836 30288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456: No bootstrap required, opened a new log
I20260810 06:24:09.035573 30288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c3cd7d8905451f92daaac3927f1456" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34165 } }
I20260810 06:24:09.035724 30294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.024142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41484 (local address 127.31.204.254:34165)
0810 06:24:09.024564 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:09.024578 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:09.024612 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:24:09.025225 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:09.025240 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:09.025549 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:09.025843 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:09.026689 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.027651 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.028486 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.028932 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.029074 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:09.031540 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:09.031545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:09.031556 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:09.031582 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:09.033104 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:09.033562 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:09.033566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:09.033570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:09.033674 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:09.033934 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:09.033936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:09.033951 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:09.034356 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:09.034506 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:09.034785 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 06:24:09.035026 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":130,"threads_started":1}
I20260810 06:24:09.035795 30288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:09.035851 30288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c3cd7d8905451f92daaac3927f1456, State: Initialized, Role: FOLLOWER
I20260810 06:24:09.036213 30288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c3cd7d8905451f92daaac3927f1456" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34165 } }
I20260810 06:24:09.036336 30288 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:09.036399 30288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:09.036496 30288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:09.037475 30288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c3cd7d8905451f92daaac3927f1456" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34165 } }
I20260810 06:24:09.037812 30288 leader_election.cc:302] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53c3cd7d8905451f92daaac3927f1456; no voters:
I20260810 06:24:09.038178 30288 leader_election.cc:288] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:09.038262 30300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:09.038468 30300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456 [term 1 LEADER]: Becoming Leader. State: Replica: 53c3cd7d8905451f92daaac3927f1456, State: Running, Role: LEADER
I20260810 06:24:09.038808 30300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c3cd7d8905451f92daaac3927f1456" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34165 } }
I20260810 06:24:09.039229 30288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:09.040227 30301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53c3cd7d8905451f92daaac3927f1456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c3cd7d8905451f92daaac3927f1456" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34165 } } }
I20260810 06:24:09.040352 30301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:09.040663 30303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53c3cd7d8905451f92daaac3927f1456. Latest consensus state: current_term: 1 leader_uuid: "53c3cd7d8905451f92daaac3927f1456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c3cd7d8905451f92daaac3927f1456" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 34165 } } }
I20260810 06:24:09.040750 30303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:09.040740 30316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:09.043907 30316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:09.048761 30316 catalog_manager.cc:1347] Generated new cluster ID: ad7cb2cf3a6b4b378247eac8c25fb603
I20260810 06:24:09.048810 30316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:09.085319 30316 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:09.086238 30316 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:09.091903 30316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456: Generated new TSK 0
I20260810 06:24:09.092494 30316 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
I20260810 06:24:09.118676 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34165
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:09.250867 30344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:09.251077 30344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:09.251129 30344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:09.254993 30344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:09.255038 30344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:09.255110 30344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:24:09.259992 30344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:09.260835 30344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:09.261727 30344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:09.270422 30349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:09.270430 30351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:09.270462 30353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:09.270562 30344 server_base.cc:1034] running on GCE node
I20260810 06:24:09.271036 30344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:09.271605 30344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:09.272754 30344 hybrid_clock.cc:648] HybridClock initialized: now 1786343049272740 us; error 27 us; skew 500 ppm
Aug 10 06:24:09 dist-test-slave-q9h9 krb5kdc[30149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343049, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:09.275245 30344 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:24:09.276005 30344 webserver.cc:460] Webserver started at http://127.31.204.193:39127/ using document root <none> and password file <none>
I20260810 06:24:09.276489 30344 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:09.276553 30344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:09.276736 30344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:09.278620 30344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "a61154ec40a84829ad918b928b8d1f7f"
format_stamp: "Formatted at 2026-08-10 06:24:09 on dist-test-slave-q9h9"
server_key: "8a2d5f866223de1333516f4dfa3034dd"
server_key_iv: "1f102e5ffc0c28681d383fa3944decdc"
server_key_version: "encryptionkey@0"
I20260810 06:24:09.279124 30344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "a61154ec40a84829ad918b928b8d1f7f"
format_stamp: "Formatted at 2026-08-10 06:24:09 on dist-test-slave-q9h9"
server_key: "8a2d5f866223de1333516f4dfa3034dd"
server_key_iv: "1f102e5ffc0c28681d383fa3944decdc"
server_key_version: "encryptionkey@0"
I20260810 06:24:09.282442 30344 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:24:09.284636 30367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:09.285741 30344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:09.285853 30344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "a61154ec40a84829ad918b928b8d1f7f"
format_stamp: "Formatted at 2026-08-10 06:24:09 on dist-test-slave-q9h9"
server_key: "8a2d5f866223de1333516f4dfa3034dd"
server_key_iv: "1f102e5ffc0c28681d383fa3944decdc"
server_key_version: "encryptionkey@0"
I20260810 06:24:09.285939 30344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:09.296602 30344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:09.299068 30344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:09.299226 30344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:09.299739 30344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:09.300470 30344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:09.300516 30344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:09.300565 30344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:09.300601 30344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:09.309788 30344 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:38097
I20260810 06:24:09.309818 30563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:38097 every 8 connection(s)
I20260810 06:24:09.310583 30344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:24:09.314486 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 30344
I20260810 06:24:09.314606 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:24:09.314885 32563 external_mini_cluster.cc:1428] Setting key a00775ac4809f439197b4567d01a1ef7
Aug 10 06:24:09 dist-test-slave-q9h9 krb5kdc[30149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343049, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:24:09.324043 30294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.312232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:36183 (local address 127.31.204.254:34165)
0810 06:24:09.312350 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:09.312354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:09.313123 (+ 769us) server_negotiation.cc:407] Connection header received
0810 06:24:09.314346 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:09.314349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:09.314405 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:09.314483 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:09.315607 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.316193 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.317642 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.317865 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.317932 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:09.320576 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:09.320580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:09.320581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:09.320608 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:09.321924 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:09.322529 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:09.322531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:09.322532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:09.322614 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:09.322975 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:09.322977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:09.322978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:09.323136 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:09.323203 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:09.323821 (+ 618us) server_negotiation.cc:299] Negotiation successful
0810 06:24:09.323926 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:24:09.325032 30572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.312553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34165 (local address 127.31.204.193:36183)
0810 06:24:09.312976 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:09.313014 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:09.314144 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:09.314676 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:09.314684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:09.315044 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:09.315480 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:09.315495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.316302 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.316305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:09.317525 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:09.317536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.317991 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.317995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:09.318195 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:09.318776 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:09.318791 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:09.320409 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:09.322065 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:09.322072 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:09.322087 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:09.322431 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:09.322693 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:09.322697 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:09.322699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:09.322891 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:09.323226 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:09.323233 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:09.323528 (+ 295us) client_negotiation.cc:769] Sending connection context
0810 06:24:09.323734 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 06:24:09.324054 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.31.204.194@KRBTEST.COM; defaulting to no policy
I20260810 06:24:09.326256 30564 heartbeater.cc:347] Connected to a master server at 127.31.204.254:34165
I20260810 06:24:09.326453 30564 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:09.326905 30564 heartbeater.cc:511] Master 127.31.204.254:34165 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.194@KRBTEST.COM" created.
I20260810 06:24:09.328397 30227 ts_manager.cc:194] Registered new tserver with Master: a61154ec40a84829ad918b928b8d1f7f (127.31.204.193:38097)
I20260810 06:24:09.329528 30227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:36183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
I20260810 06:24:09.341691 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34165
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:09.448709 30578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:09.448937 30578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:09.448976 30578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:09.452782 30578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:09.452834 30578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:09.452889 30578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:24:09.457826 30578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:09.458734 30578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:09.459642 30578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:09.468916 30590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:09.468961 30578 server_base.cc:1034] running on GCE node
W20260810 06:24:09.468911 30593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:09.468973 30595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:09.469491 30578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:09.470144 30578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:09.471312 30578 hybrid_clock.cc:648] HybridClock initialized: now 1786343049471292 us; error 30 us; skew 500 ppm
Aug 10 06:24:09 dist-test-slave-q9h9 krb5kdc[30149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343049, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:09.473842 30578 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:24:09.474699 30578 webserver.cc:460] Webserver started at http://127.31.204.194:43529/ using document root <none> and password file <none>
I20260810 06:24:09.475227 30578 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:09.475296 30578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:09.475489 30578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:09.477460 30578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "e909fd0c19204c83b357326baba3c8d2"
format_stamp: "Formatted at 2026-08-10 06:24:09 on dist-test-slave-q9h9"
server_key: "ad34ad21c7444a87801f24e82c80dee8"
server_key_iv: "45e3e094f4588bed59f55c9ba54d53e0"
server_key_version: "encryptionkey@0"
I20260810 06:24:09.477990 30578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "e909fd0c19204c83b357326baba3c8d2"
format_stamp: "Formatted at 2026-08-10 06:24:09 on dist-test-slave-q9h9"
server_key: "ad34ad21c7444a87801f24e82c80dee8"
server_key_iv: "45e3e094f4588bed59f55c9ba54d53e0"
server_key_version: "encryptionkey@0"
I20260810 06:24:09.482395 30578 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 06:24:09.484700 30611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:09.485833 30578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:09.485929 30578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "e909fd0c19204c83b357326baba3c8d2"
format_stamp: "Formatted at 2026-08-10 06:24:09 on dist-test-slave-q9h9"
server_key: "ad34ad21c7444a87801f24e82c80dee8"
server_key_iv: "45e3e094f4588bed59f55c9ba54d53e0"
server_key_version: "encryptionkey@0"
I20260810 06:24:09.486006 30578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:09.504140 30578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:09.506613 30578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:09.506752 30578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:09.507248 30578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:09.507998 30578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:09.508045 30578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:09.508082 30578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:09.508103 30578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:09.516181 30578 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:44495
I20260810 06:24:09.516254 30797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:44495 every 8 connection(s)
I20260810 06:24:09.517006 30578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:24:09.517338 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 30578
I20260810 06:24:09.517467 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:24:09.517802 32563 external_mini_cluster.cc:1428] Setting key 871e870bed6e60adaa350ec206aaf4c2
Aug 10 06:24:09 dist-test-slave-q9h9 krb5kdc[30149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343049, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.204.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.195@KRBTEST.COM" created.
I20260810 06:24:09.530221 30294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.518801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:33567 (local address 127.31.204.254:34165)
0810 06:24:09.518928 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:09.518933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:09.519856 (+ 923us) server_negotiation.cc:407] Connection header received
0810 06:24:09.521140 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:09.521143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:09.521200 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:09.521270 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:09.522444 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.522953 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.524060 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.524280 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.524369 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:09.526903 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:09.526906 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:09.526908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:09.526933 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:09.528246 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:09.528718 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:09.528722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:09.528723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:09.528799 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:09.529104 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:09.529108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:09.529110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:09.529319 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:09.529440 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:09.529957 (+ 517us) server_negotiation.cc:299] Negotiation successful
0810 06:24:09.530103 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:24:09.531301 30805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.519216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34165 (local address 127.31.204.194:33567)
0810 06:24:09.519722 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:09.519763 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:09.520922 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:09.521453 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:09.521460 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:09.521794 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:09.522301 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:09.522322 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.523072 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.523075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:09.523939 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:09.523948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.524432 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.524437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:09.524587 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:09.525161 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:09.525175 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:09.526749 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:09.528358 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:09.528364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:09.528379 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:09.528625 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:09.528881 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:09.528884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:09.528886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:09.529017 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:09.529458 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:09.529464 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:09.529702 (+ 238us) client_negotiation.cc:769] Sending connection context
0810 06:24:09.529882 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 06:24:09.530233 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":138,"threads_started":1}
I20260810 06:24:09.532522 30800 heartbeater.cc:347] Connected to a master server at 127.31.204.254:34165
I20260810 06:24:09.532739 30800 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:09.533185 30800 heartbeater.cc:511] Master 127.31.204.254:34165 requested a full tablet report, sending...
I20260810 06:24:09.534250 30227 ts_manager.cc:194] Registered new tserver with Master: e909fd0c19204c83b357326baba3c8d2 (127.31.204.194:44495)
I20260810 06:24:09.535010 30227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:33567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
I20260810 06:24:09.544021 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34165
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:09.657691 30808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:09.657935 30808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:09.657982 30808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:09.661895 30808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:09.661953 30808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:09.662047 30808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:24:09.667135 30808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:34165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:09.668090 30808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:09.669092 30808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:24:09.678346 30808 server_base.cc:1034] running on GCE node
W20260810 06:24:09.678306 30818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:09.678364 30819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:09.678581 30823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:09.678970 30808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:09.679551 30808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:09.680706 30808 hybrid_clock.cc:648] HybridClock initialized: now 1786343049680685 us; error 34 us; skew 500 ppm
Aug 10 06:24:09 dist-test-slave-q9h9 krb5kdc[30149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343049, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:09.683276 30808 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:24:09.684129 30808 webserver.cc:460] Webserver started at http://127.31.204.195:41977/ using document root <none> and password file <none>
I20260810 06:24:09.684633 30808 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:09.684700 30808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:09.684899 30808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:09.686873 30808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "f1c6c06f4ac54869b8f13251086e6bbd"
format_stamp: "Formatted at 2026-08-10 06:24:09 on dist-test-slave-q9h9"
server_key: "319b2dbbb276a0d4183a5ef1c1ccb7aa"
server_key_iv: "5941999a85f6b2522114e75fd6bfb7d6"
server_key_version: "encryptionkey@0"
I20260810 06:24:09.687395 30808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "f1c6c06f4ac54869b8f13251086e6bbd"
format_stamp: "Formatted at 2026-08-10 06:24:09 on dist-test-slave-q9h9"
server_key: "319b2dbbb276a0d4183a5ef1c1ccb7aa"
server_key_iv: "5941999a85f6b2522114e75fd6bfb7d6"
server_key_version: "encryptionkey@0"
I20260810 06:24:09.691043 30808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:24:09.693235 30839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:09.694427 30808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:24:09.694527 30808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "f1c6c06f4ac54869b8f13251086e6bbd"
format_stamp: "Formatted at 2026-08-10 06:24:09 on dist-test-slave-q9h9"
server_key: "319b2dbbb276a0d4183a5ef1c1ccb7aa"
server_key_iv: "5941999a85f6b2522114e75fd6bfb7d6"
server_key_version: "encryptionkey@0"
I20260810 06:24:09.694612 30808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:09.711186 30808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:09.713614 30808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:09.713778 30808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:09.714347 30808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:09.715121 30808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:09.715167 30808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:09.715230 30808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:09.715257 30808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:09.723361 30808 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:39657
I20260810 06:24:09.723387 31041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:39657 every 8 connection(s)
I20260810 06:24:09.724165 30808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:24:09.729744 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 30808
I20260810 06:24:09.729884 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:24:09.730193 32563 external_mini_cluster.cc:1428] Setting key 1bb10791985c8afe321074dbebe69d80
Aug 10 06:24:09 dist-test-slave-q9h9 krb5kdc[30149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343049, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:24:09.737121 30294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.726051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:34623 (local address 127.31.204.254:34165)
0810 06:24:09.726183 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:09.726188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:09.727008 (+ 820us) server_negotiation.cc:407] Connection header received
0810 06:24:09.728203 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:09.728206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:09.728252 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:09.728323 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:09.729365 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.730101 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.731423 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.731701 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.731788 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:09.734090 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:09.734094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:09.734096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:09.734124 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:09.735274 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:09.735734 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:09.735738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:09.735740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:09.735822 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:09.736115 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:09.736119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:09.736121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:09.736303 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:09.736387 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:09.736847 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 06:24:09.736965 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 06:24:09.737843 31048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.726434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:34165 (local address 127.31.204.195:34623)
0810 06:24:09.726864 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:09.726904 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:09.728026 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:09.728464 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:09.728470 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:09.728790 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:09.729254 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:09.729268 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.730238 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.730241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:09.731308 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:09.731320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.731842 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.731846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:09.732038 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:09.732563 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:09.732579 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:09.733918 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:09.735377 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:09.735384 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:09.735397 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:09.735633 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:09.735899 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:09.735902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:09.735904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:09.736029 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:09.736399 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:09.736406 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:09.736627 (+ 221us) client_negotiation.cc:769] Sending connection context
0810 06:24:09.736784 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 06:24:09.737078 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":102,"threads_started":1}
I20260810 06:24:09.738833 31045 heartbeater.cc:347] Connected to a master server at 127.31.204.254:34165
I20260810 06:24:09.739043 31045 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:09.739511 31045 heartbeater.cc:511] Master 127.31.204.254:34165 requested a full tablet report, sending...
I20260810 06:24:09.740463 30227 ts_manager.cc:194] Registered new tserver with Master: f1c6c06f4ac54869b8f13251086e6bbd (127.31.204.195:39657)
I20260810 06:24:09.741048 30227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:34623
I20260810 06:24:09.743758 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:24:09.751883 30294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.745550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.31.204.254:34165)
0810 06:24:09.745655 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:09.745658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:09.745855 (+ 197us) server_negotiation.cc:407] Connection header received
0810 06:24:09.745945 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:09.745947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:09.745983 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:09.746074 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:09.746697 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.747158 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.747905 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.748100 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.748162 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:09.749447 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:09.749449 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:09.749451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:09.749470 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:09.750556 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:09.750976 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:09.750978 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:09.750980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:09.751044 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:09.751257 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:09.751259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:09.751260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:09.751433 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:09.751529 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:09.751710 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 06:24:09.751780 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:24:09.754287 30227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:24:09.757126 30227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:24:09.771127 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.765140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58626 (local address 127.31.204.193:38097)
0810 06:24:09.765639 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:09.765653 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:09.765680 (+ 27us) server_negotiation.cc:407] Connection header received
0810 06:24:09.765762 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:09.765768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:09.765982 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:09.766177 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:09.766602 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.767680 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.769373 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.770204 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.770328 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:09.770448 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:09.770909 (+ 461us) server_negotiation.cc:299] Negotiation successful
0810 06:24:09.771001 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":131,"threads_started":1}
I20260810 06:24:09.771162 31065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.765303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:39657 (local address 127.0.0.1:42046)
0810 06:24:09.766229 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:09.766241 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:09.766343 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:09.766771 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:09.766774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:09.766794 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:09.766905 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:09.766914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.768237 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.768241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:09.769200 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:09.769210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.770514 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.770518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:09.770688 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:09.770739 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 06:24:09.770843 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 06:24:09.770958 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":52,"thread_start_us":95,"threads_started":1}
I20260810 06:24:09.771462 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.765941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41352 (local address 127.31.204.194:44495)
0810 06:24:09.766774 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:09.766789 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:09.766811 (+ 22us) server_negotiation.cc:407] Connection header received
0810 06:24:09.767038 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:09.767043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:09.767217 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:09.767355 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:09.767706 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.768613 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.769873 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.770713 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.770838 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:09.770965 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:09.771232 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 06:24:09.771318 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":170,"threads_started":1}
I20260810 06:24:09.771479 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.765724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42046 (local address 127.31.204.195:39657)
0810 06:24:09.766207 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:09.766220 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:09.766341 (+ 121us) server_negotiation.cc:407] Connection header received
0810 06:24:09.766447 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:09.766452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:09.766643 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:09.766784 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:09.767050 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.768089 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.769332 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.770222 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.770323 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:09.770448 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:09.771061 (+ 613us) server_negotiation.cc:299] Negotiation successful
0810 06:24:09.771157 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":165,"threads_started":1}
I20260810 06:24:09.771533 31060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.764958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:38097 (local address 127.0.0.1:58626)
0810 06:24:09.765402 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:09.765447 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:09.765582 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:09.766326 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:09.766331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:09.766350 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:09.766480 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:09.766486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.767907 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.767910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:09.769205 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:09.769218 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.770514 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.770518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:09.770695 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:09.770743 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 06:24:09.771335 (+ 592us) client_negotiation.cc:240] Negotiation successful
0810 06:24:09.771412 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":171,"thread_start_us":73,"threads_started":1}
I20260810 06:24:09.771733 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.765622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:44495 (local address 127.0.0.1:41352)
0810 06:24:09.766450 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:09.766464 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:09.766567 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:09.767499 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:09.767502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:09.767519 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:09.767595 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:09.767600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.768759 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.768763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:09.769721 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:09.769732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.770848 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.770852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:09.770942 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:09.770968 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 06:24:09.771047 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 06:24:09.771150 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":5,"thread_start_us":90,"threads_started":1}
I20260810 06:24:09.774081 30917 tablet_service.cc:1467] Processing CreateTablet for tablet 3b464f1de3d449e28e79d745bcc0db68 (DEFAULT_TABLE table=test-table [id=28e718384a684e05b2b13f25d2520e60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:24:09.774334 30444 tablet_service.cc:1467] Processing CreateTablet for tablet 3b464f1de3d449e28e79d745bcc0db68 (DEFAULT_TABLE table=test-table [id=28e718384a684e05b2b13f25d2520e60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:24:09.774740 30686 tablet_service.cc:1467] Processing CreateTablet for tablet 3b464f1de3d449e28e79d745bcc0db68 (DEFAULT_TABLE table=test-table [id=28e718384a684e05b2b13f25d2520e60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:24:09.775214 30917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b464f1de3d449e28e79d745bcc0db68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:09.775393 30444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b464f1de3d449e28e79d745bcc0db68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:09.775704 30686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b464f1de3d449e28e79d745bcc0db68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:09.781178 31072 tablet_bootstrap.cc:492] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd: Bootstrap starting.
I20260810 06:24:09.782516 31077 tablet_bootstrap.cc:492] T 3b464f1de3d449e28e79d745bcc0db68 P a61154ec40a84829ad918b928b8d1f7f: Bootstrap starting.
I20260810 06:24:09.783401 31078 tablet_bootstrap.cc:492] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2: Bootstrap starting.
I20260810 06:24:09.784124 31072 tablet_bootstrap.cc:654] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:09.785002 31077 tablet_bootstrap.cc:654] T 3b464f1de3d449e28e79d745bcc0db68 P a61154ec40a84829ad918b928b8d1f7f: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:09.785247 31078 tablet_bootstrap.cc:654] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:09.785449 31072 log.cc:824] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:09.786180 31077 log.cc:824] T 3b464f1de3d449e28e79d745bcc0db68 P a61154ec40a84829ad918b928b8d1f7f: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:09.786422 31078 log.cc:824] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:09.787506 31072 tablet_bootstrap.cc:492] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd: No bootstrap required, opened a new log
I20260810 06:24:09.787683 31072 ts_tablet_manager.cc:1397] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 06:24:09.788172 31077 tablet_bootstrap.cc:492] T 3b464f1de3d449e28e79d745bcc0db68 P a61154ec40a84829ad918b928b8d1f7f: No bootstrap required, opened a new log
I20260810 06:24:09.788250 31078 tablet_bootstrap.cc:492] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2: No bootstrap required, opened a new log
I20260810 06:24:09.788378 31077 ts_tablet_manager.cc:1397] T 3b464f1de3d449e28e79d745bcc0db68 P a61154ec40a84829ad918b928b8d1f7f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:24:09.788424 31078 ts_tablet_manager.cc:1397] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:24:09.791026 31072 raft_consensus.cc:348] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61154ec40a84829ad918b928b8d1f7f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38097 } } peers { permanent_uuid: "e909fd0c19204c83b357326baba3c8d2" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44495 } } peers { permanent_uuid: "f1c6c06f4ac54869b8f13251086e6bbd" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39657 } }
I20260810 06:24:09.791229 31072 raft_consensus.cc:374] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:09.791289 31072 raft_consensus.cc:729] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1c6c06f4ac54869b8f13251086e6bbd, State: Initialized, Role: FOLLOWER
I20260810 06:24:09.791676 31072 consensus_queue.cc:260] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61154ec40a84829ad918b928b8d1f7f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38097 } } peers { permanent_uuid: "e909fd0c19204c83b357326baba3c8d2" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44495 } } peers { permanent_uuid: "f1c6c06f4ac54869b8f13251086e6bbd" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39657 } }
I20260810 06:24:09.792269 31045 heartbeater.cc:503] Master 127.31.204.254:34165 was elected leader, sending a full tablet report...
I20260810 06:24:09.792296 31078 raft_consensus.cc:348] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61154ec40a84829ad918b928b8d1f7f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38097 } } peers { permanent_uuid: "e909fd0c19204c83b357326baba3c8d2" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44495 } } peers { permanent_uuid: "f1c6c06f4ac54869b8f13251086e6bbd" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39657 } }
I20260810 06:24:09.792296 31077 raft_consensus.cc:348] T 3b464f1de3d449e28e79d745bcc0db68 P a61154ec40a84829ad918b928b8d1f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61154ec40a84829ad918b928b8d1f7f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38097 } } peers { permanent_uuid: "e909fd0c19204c83b357326baba3c8d2" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44495 } } peers { permanent_uuid: "f1c6c06f4ac54869b8f13251086e6bbd" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39657 } }
I20260810 06:24:09.792554 31078 raft_consensus.cc:374] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:09.792564 31077 raft_consensus.cc:374] T 3b464f1de3d449e28e79d745bcc0db68 P a61154ec40a84829ad918b928b8d1f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:09.792619 31078 raft_consensus.cc:729] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e909fd0c19204c83b357326baba3c8d2, State: Initialized, Role: FOLLOWER
I20260810 06:24:09.792629 31077 raft_consensus.cc:729] T 3b464f1de3d449e28e79d745bcc0db68 P a61154ec40a84829ad918b928b8d1f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a61154ec40a84829ad918b928b8d1f7f, State: Initialized, Role: FOLLOWER
I20260810 06:24:09.793051 31077 consensus_queue.cc:260] T 3b464f1de3d449e28e79d745bcc0db68 P a61154ec40a84829ad918b928b8d1f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61154ec40a84829ad918b928b8d1f7f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38097 } } peers { permanent_uuid: "e909fd0c19204c83b357326baba3c8d2" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44495 } } peers { permanent_uuid: "f1c6c06f4ac54869b8f13251086e6bbd" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39657 } }
I20260810 06:24:09.793051 31078 consensus_queue.cc:260] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61154ec40a84829ad918b928b8d1f7f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38097 } } peers { permanent_uuid: "e909fd0c19204c83b357326baba3c8d2" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44495 } } peers { permanent_uuid: "f1c6c06f4ac54869b8f13251086e6bbd" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39657 } }
I20260810 06:24:09.793581 31072 ts_tablet_manager.cc:1428] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 06:24:09.794037 31077 ts_tablet_manager.cc:1428] T 3b464f1de3d449e28e79d745bcc0db68 P a61154ec40a84829ad918b928b8d1f7f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 06:24:09.794037 31078 ts_tablet_manager.cc:1428] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 06:24:09.794377 30564 heartbeater.cc:503] Master 127.31.204.254:34165 was elected leader, sending a full tablet report...
I20260810 06:24:09.794377 30800 heartbeater.cc:503] Master 127.31.204.254:34165 was elected leader, sending a full tablet report...
W20260810 06:24:09.811640 30566 tablet.cc:2369] T 3b464f1de3d449e28e79d745bcc0db68 P a61154ec40a84829ad918b928b8d1f7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:24:09.819093 31087 raft_consensus.cc:482] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:24:09.819242 31087 raft_consensus.cc:504] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61154ec40a84829ad918b928b8d1f7f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38097 } } peers { permanent_uuid: "e909fd0c19204c83b357326baba3c8d2" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44495 } } peers { permanent_uuid: "f1c6c06f4ac54869b8f13251086e6bbd" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39657 } }
I20260810 06:24:09.820245 31087 leader_election.cc:288] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a61154ec40a84829ad918b928b8d1f7f (127.31.204.193:38097), f1c6c06f4ac54869b8f13251086e6bbd (127.31.204.195:39657)
I20260810 06:24:09.823779 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.820552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:35917 (local address 127.31.204.193:38097)
0810 06:24:09.820663 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:09.820668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:09.820682 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:24:09.820731 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:09.820734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:09.820782 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:09.820861 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:09.821151 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.821746 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.822849 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.823252 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.823344 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:09.823389 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:09.823588 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 06:24:09.823659 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:24:09.823803 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.820552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:58489 (local address 127.31.204.195:39657)
0810 06:24:09.820663 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:09.820668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:09.820933 (+ 265us) server_negotiation.cc:407] Connection header received
0810 06:24:09.821052 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:09.821055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:09.821104 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:09.821170 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:09.821417 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.821909 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.822934 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.823321 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.823431 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:09.823478 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:09.823626 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 06:24:09.823689 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:24:09.823819 30805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.820436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:38097 (local address 127.31.204.194:35917)
0810 06:24:09.820535 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:09.820549 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:09.820630 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:09.820888 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:09.820890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:09.820910 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:09.821043 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:09.821050 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.821844 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.821847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:09.822712 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:09.822721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.823365 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.823368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:09.823468 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:09.823489 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 06:24:09.823556 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 06:24:09.823626 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"spinlock_wait_cycles":21376}
I20260810 06:24:09.823824 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.820436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:39657 (local address 127.31.204.194:58489)
0810 06:24:09.820841 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:09.820855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:09.820963 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:09.821188 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:09.821191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:09.821209 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:09.821326 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:09.821332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.822036 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.822040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:09.822815 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:09.822826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.823435 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.823438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:09.823509 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:09.823526 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 06:24:09.823593 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 06:24:09.823659 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":104,"thread_start_us":119,"threads_started":1}
I20260810 06:24:09.824440 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b464f1de3d449e28e79d745bcc0db68" candidate_uuid: "e909fd0c19204c83b357326baba3c8d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1c6c06f4ac54869b8f13251086e6bbd" is_pre_election: true
I20260810 06:24:09.824440 30484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b464f1de3d449e28e79d745bcc0db68" candidate_uuid: "e909fd0c19204c83b357326baba3c8d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a61154ec40a84829ad918b928b8d1f7f" is_pre_election: true
I20260810 06:24:09.824708 30956 raft_consensus.cc:2454] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e909fd0c19204c83b357326baba3c8d2 in term 0.
I20260810 06:24:09.824708 30484 raft_consensus.cc:2454] T 3b464f1de3d449e28e79d745bcc0db68 P a61154ec40a84829ad918b928b8d1f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e909fd0c19204c83b357326baba3c8d2 in term 0.
I20260810 06:24:09.825143 30617 leader_election.cc:302] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a61154ec40a84829ad918b928b8d1f7f, e909fd0c19204c83b357326baba3c8d2, f1c6c06f4ac54869b8f13251086e6bbd; no voters:
I20260810 06:24:09.825379 31087 raft_consensus.cc:2781] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:24:09.825460 31087 raft_consensus.cc:482] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:24:09.825510 31087 raft_consensus.cc:3037] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:09.826545 31087 raft_consensus.cc:504] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61154ec40a84829ad918b928b8d1f7f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38097 } } peers { permanent_uuid: "e909fd0c19204c83b357326baba3c8d2" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44495 } } peers { permanent_uuid: "f1c6c06f4ac54869b8f13251086e6bbd" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39657 } }
I20260810 06:24:09.826912 31087 leader_election.cc:288] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [CANDIDATE]: Term 1 election: Requested vote from peers a61154ec40a84829ad918b928b8d1f7f (127.31.204.193:38097), f1c6c06f4ac54869b8f13251086e6bbd (127.31.204.195:39657)
I20260810 06:24:09.827205 30484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b464f1de3d449e28e79d745bcc0db68" candidate_uuid: "e909fd0c19204c83b357326baba3c8d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a61154ec40a84829ad918b928b8d1f7f"
I20260810 06:24:09.827219 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b464f1de3d449e28e79d745bcc0db68" candidate_uuid: "e909fd0c19204c83b357326baba3c8d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1c6c06f4ac54869b8f13251086e6bbd"
I20260810 06:24:09.827324 30484 raft_consensus.cc:3037] T 3b464f1de3d449e28e79d745bcc0db68 P a61154ec40a84829ad918b928b8d1f7f [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:09.827327 30956 raft_consensus.cc:3037] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:09.828285 30484 raft_consensus.cc:2454] T 3b464f1de3d449e28e79d745bcc0db68 P a61154ec40a84829ad918b928b8d1f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e909fd0c19204c83b357326baba3c8d2 in term 1.
I20260810 06:24:09.828291 30956 raft_consensus.cc:2454] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e909fd0c19204c83b357326baba3c8d2 in term 1.
I20260810 06:24:09.828555 30617 leader_election.cc:302] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e909fd0c19204c83b357326baba3c8d2, f1c6c06f4ac54869b8f13251086e6bbd; no voters:
I20260810 06:24:09.828727 31087 raft_consensus.cc:2781] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:09.828912 31087 raft_consensus.cc:686] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [term 1 LEADER]: Becoming Leader. State: Replica: e909fd0c19204c83b357326baba3c8d2, State: Running, Role: LEADER
I20260810 06:24:09.829181 31087 consensus_queue.cc:237] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61154ec40a84829ad918b928b8d1f7f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38097 } } peers { permanent_uuid: "e909fd0c19204c83b357326baba3c8d2" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44495 } } peers { permanent_uuid: "f1c6c06f4ac54869b8f13251086e6bbd" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39657 } }
I20260810 06:24:09.831677 30227 catalog_manager.cc:5495] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e909fd0c19204c83b357326baba3c8d2 (127.31.204.194). New cstate: current_term: 1 leader_uuid: "e909fd0c19204c83b357326baba3c8d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61154ec40a84829ad918b928b8d1f7f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e909fd0c19204c83b357326baba3c8d2" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1c6c06f4ac54869b8f13251086e6bbd" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39657 } health_report { overall_health: UNKNOWN } } }
I20260810 06:24:09.849483 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.846491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41360 (local address 127.31.204.194:44495)
0810 06:24:09.846595 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:09.846598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:09.846609 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:24:09.846732 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:09.846734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:09.846790 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:09.846867 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:24:09.847043 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.847552 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.848320 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:09.848518 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:09.848582 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:09.848837 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:24:09.849189 (+ 352us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:24:09.849293 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:09.849354 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 06:24:09.849385 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 06:24:09.854877 30956 raft_consensus.cc:1261] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd [term 1 FOLLOWER]: Refusing update from remote peer e909fd0c19204c83b357326baba3c8d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:24:09.854877 30484 raft_consensus.cc:1261] T 3b464f1de3d449e28e79d745bcc0db68 P a61154ec40a84829ad918b928b8d1f7f [term 1 FOLLOWER]: Refusing update from remote peer e909fd0c19204c83b357326baba3c8d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:24:09.855541 31087 consensus_queue.cc:1035] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a61154ec40a84829ad918b928b8d1f7f" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:24:09.855732 31092 consensus_queue.cc:1035] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1c6c06f4ac54869b8f13251086e6bbd" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:24:09.864601 31099 mvcc.cc:205] Tried to move back new op lower bound from 7316861132198731776 to 7316861132102541312. Current Snapshot: MvccSnapshot[applied={T|T < 7316861132198731776}]
I20260810 06:24:09.870004 30223 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:24:09.870172 30223 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:24:09.872284 30223 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53c3cd7d8905451f92daaac3927f1456: Sending DeleteTablet for 3 replicas of tablet 3b464f1de3d449e28e79d745bcc0db68
I20260810 06:24:09.873018 30917 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b464f1de3d449e28e79d745bcc0db68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:24:09 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:42046
I20260810 06:24:09.873018 30444 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b464f1de3d449e28e79d745bcc0db68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:24:09 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:58626
I20260810 06:24:09.873445 30686 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b464f1de3d449e28e79d745bcc0db68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:24:09 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:41352
I20260810 06:24:09.873646 31116 tablet_replica.cc:330] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd: stopping tablet replica
I20260810 06:24:09.873803 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 30344
I20260810 06:24:09.873922 31116 raft_consensus.cc:2229] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:24:09.874087 31115 tablet_replica.cc:330] T 3b464f1de3d449e28e79d745bcc0db68 P a61154ec40a84829ad918b928b8d1f7f: stopping tablet replica
I20260810 06:24:09.874646 31116 raft_consensus.cc:2258] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:24:09.882200 31117 tablet_replica.cc:330] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2: stopping tablet replica
I20260810 06:24:09.882283 31116 ts_tablet_manager.cc:1905] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:24:09.882596 31117 raft_consensus.cc:2229] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:24:09.882995 31117 raft_consensus.cc:2258] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:24:09.884939 31116 ts_tablet_manager.cc:1918] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:24:09.885030 31116 log.cc:1196] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/3b464f1de3d449e28e79d745bcc0db68
I20260810 06:24:09.885383 31116 ts_tablet_manager.cc:1939] T 3b464f1de3d449e28e79d745bcc0db68 P f1c6c06f4ac54869b8f13251086e6bbd: Deleting consensus metadata
I20260810 06:24:09.886572 30202 catalog_manager.cc:4841] TS f1c6c06f4ac54869b8f13251086e6bbd (127.31.204.195:39657): tablet 3b464f1de3d449e28e79d745bcc0db68 (table test-table [id=28e718384a684e05b2b13f25d2520e60]) successfully deleted
I20260810 06:24:09.887465 31117 ts_tablet_manager.cc:1905] T 3b464f1de3d449e28e79d745bcc0db68 P e909fd0c19204c83b357326baba3c8d2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 06:24:09.887902 30201 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.204.193:38097 (error 108)
I20260810 06:24:09.888820 31060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.888558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:38097 (local address 127.0.0.1:58636)
0810 06:24:09.888676 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:09.888740 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:38097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 06:24:09.888942 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 30578
W20260810 06:24:09.897035 30201 catalog_manager.cc:4584] TS a61154ec40a84829ad918b928b8d1f7f (127.31.204.193:38097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b464f1de3d449e28e79d745bcc0db68: Network error: Client connection negotiation failed: client connection to 127.31.204.193:38097: connect: Connection refused (error 111)
I20260810 06:24:09.902346 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 30808
I20260810 06:24:09.909736 31060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:09.902554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:44495 (local address 127.0.0.1:41372)
0810 06:24:09.909585 (+ 7031us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:09.909647 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.194:44495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6929}
I20260810 06:24:09.917922 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 30168
2026-08-10T06:24:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2340 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T06:24:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:09Z Disabled control of system clock
I20260810 06:24:09.950548 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:41917
--webserver_interface=127.31.204.254
--webserver_port=0
--builtin_ntp_servers=127.31.204.212:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:41917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 06:24:10.058928 31123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:10.059177 31123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:10.059221 31123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:10.063210 31123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:10.063270 31123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:10.063289 31123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:10.063313 31123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:10.063336 31123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:10.068085 31123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:41917
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:41917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:10.069051 31123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:10.070396 31123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:10.078284 31131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:10.078306 31123 server_base.cc:1034] running on GCE node
W20260810 06:24:10.078238 31135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:10.078255 31133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:10.079028 31123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:10.080052 31123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:10.081208 31123 hybrid_clock.cc:648] HybridClock initialized: now 1786343050081187 us; error 34 us; skew 500 ppm
I20260810 06:24:10.082953 31123 webserver.cc:460] Webserver started at http://127.31.204.254:39029/ using document root <none> and password file <none>
I20260810 06:24:10.083477 31123 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:10.083540 31123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:10.083717 31123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:10.085633 31123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "5cc05db9f7a546f1a2114246b7f1c894"
format_stamp: "Formatted at 2026-08-10 06:24:10 on dist-test-slave-q9h9"
server_key: "bd30ed6fcf9c06373063c0222f972df7"
server_key_iv: "1760bee2b4e4051cf54b1ec6f5cb4da2"
server_key_version: "encryptionkey@0"
I20260810 06:24:10.086177 31123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "5cc05db9f7a546f1a2114246b7f1c894"
format_stamp: "Formatted at 2026-08-10 06:24:10 on dist-test-slave-q9h9"
server_key: "bd30ed6fcf9c06373063c0222f972df7"
server_key_iv: "1760bee2b4e4051cf54b1ec6f5cb4da2"
server_key_version: "encryptionkey@0"
I20260810 06:24:10.089488 31123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:24:10.091763 31150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:10.093050 31123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 06:24:10.093192 31123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "5cc05db9f7a546f1a2114246b7f1c894"
format_stamp: "Formatted at 2026-08-10 06:24:10 on dist-test-slave-q9h9"
server_key: "bd30ed6fcf9c06373063c0222f972df7"
server_key_iv: "1760bee2b4e4051cf54b1ec6f5cb4da2"
server_key_version: "encryptionkey@0"
I20260810 06:24:10.093284 31123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:10.110690 31123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:10.111420 31123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:10.111598 31123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:10.118136 31123 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:41917
I20260810 06:24:10.118150 31246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:41917 every 8 connection(s)
I20260810 06:24:10.119028 31123 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:10.121925 31251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:10.125349 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 31123
I20260810 06:24:10.125476 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:10.125823 32563 external_mini_cluster.cc:1428] Setting key 971ac745e5b62c1d1a49ea0805bd07dd
I20260810 06:24:10.127627 31251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894: Bootstrap starting.
I20260810 06:24:10.129715 31251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:10.130445 31251 log.cc:824] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:10.132102 31251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894: No bootstrap required, opened a new log
I20260810 06:24:10.134536 31257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:10.127263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.31.204.254:41917)
0810 06:24:10.127885 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:10.127900 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:10.127935 (+ 35us) server_negotiation.cc:407] Connection header received
0810 06:24:10.128538 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:10.128552 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:10.128922 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:10.129206 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:10.129613 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.130636 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.131493 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.131941 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.132078 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:10.132832 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:10.132836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:10.132848 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:10.133187 (+ 339us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:10.133236 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:10.133255 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:10.133338 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:10.133470 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 06:24:10.133695 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":128,"threads_started":1}
I20260810 06:24:10.135037 31251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc05db9f7a546f1a2114246b7f1c894" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 41917 } }
I20260810 06:24:10.135236 31251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:10.135295 31251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc05db9f7a546f1a2114246b7f1c894, State: Initialized, Role: FOLLOWER
I20260810 06:24:10.135702 31251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc05db9f7a546f1a2114246b7f1c894" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 41917 } }
I20260810 06:24:10.135830 31251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:10.135891 31251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:10.135985 31251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:10.136852 31251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc05db9f7a546f1a2114246b7f1c894" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 41917 } }
I20260810 06:24:10.137143 31251 leader_election.cc:302] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cc05db9f7a546f1a2114246b7f1c894; no voters:
I20260810 06:24:10.137396 31251 leader_election.cc:288] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:10.137494 31259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:10.137683 31259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894 [term 1 LEADER]: Becoming Leader. State: Replica: 5cc05db9f7a546f1a2114246b7f1c894, State: Running, Role: LEADER
I20260810 06:24:10.137974 31259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc05db9f7a546f1a2114246b7f1c894" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 41917 } }
I20260810 06:24:10.138466 31251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:10.139010 31260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cc05db9f7a546f1a2114246b7f1c894. Latest consensus state: current_term: 1 leader_uuid: "5cc05db9f7a546f1a2114246b7f1c894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc05db9f7a546f1a2114246b7f1c894" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 41917 } } }
I20260810 06:24:10.139132 31260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:10.139459 31263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cc05db9f7a546f1a2114246b7f1c894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc05db9f7a546f1a2114246b7f1c894" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 41917 } } }
I20260810 06:24:10.139559 31263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:10.140657 31273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:10.143545 31273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:10.148342 31273 catalog_manager.cc:1347] Generated new cluster ID: 165159909bfe433c8e6f8986abe03b35
I20260810 06:24:10.148401 31273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:10.170637 31273 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:10.171402 31273 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:10.181120 31273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894: Generated new TSK 0
I20260810 06:24:10.181746 31273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:10.191293 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:41917
--builtin_ntp_servers=127.31.204.212:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 06:24:10.297588 31296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:10.297842 31296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:10.297879 31296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:10.301740 31296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:10.301785 31296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:10.301849 31296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:24:10.306351 31296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:10.307196 31296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:10.308379 31296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:24:10.316943 31296 server_base.cc:1034] running on GCE node
W20260810 06:24:10.316893 31304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:10.316890 31305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:10.316893 31308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:10.317500 31296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:10.318066 31296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:10.319248 31296 hybrid_clock.cc:648] HybridClock initialized: now 1786343050319238 us; error 33 us; skew 500 ppm
I20260810 06:24:10.320814 31296 webserver.cc:460] Webserver started at http://127.31.204.193:43565/ using document root <none> and password file <none>
I20260810 06:24:10.321313 31296 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:10.321376 31296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:10.321632 31296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:10.323485 31296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "9827dc4662304358af1205edfb1c3984"
format_stamp: "Formatted at 2026-08-10 06:24:10 on dist-test-slave-q9h9"
server_key: "4d6ddc14e71c335f62bbe5b92e44f260"
server_key_iv: "e3de96dbeb6f7f6cb10e4cc4b2421aa8"
server_key_version: "encryptionkey@0"
I20260810 06:24:10.323989 31296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "9827dc4662304358af1205edfb1c3984"
format_stamp: "Formatted at 2026-08-10 06:24:10 on dist-test-slave-q9h9"
server_key: "4d6ddc14e71c335f62bbe5b92e44f260"
server_key_iv: "e3de96dbeb6f7f6cb10e4cc4b2421aa8"
server_key_version: "encryptionkey@0"
I20260810 06:24:10.327311 31296 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:24:10.329385 31315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:10.330464 31296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:10.330564 31296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "9827dc4662304358af1205edfb1c3984"
format_stamp: "Formatted at 2026-08-10 06:24:10 on dist-test-slave-q9h9"
server_key: "4d6ddc14e71c335f62bbe5b92e44f260"
server_key_iv: "e3de96dbeb6f7f6cb10e4cc4b2421aa8"
server_key_version: "encryptionkey@0"
I20260810 06:24:10.330647 31296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:10.352563 31296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:10.353067 31296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:10.353216 31296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:10.353682 31296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:10.354508 31296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:10.354553 31296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:10.354601 31296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:10.354631 31296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:10.362510 31296 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:40265
I20260810 06:24:10.362528 31501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:40265 every 8 connection(s)
I20260810 06:24:10.363405 31296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:24:10.367470 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 31296
I20260810 06:24:10.367612 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:24:10.367918 32563 external_mini_cluster.cc:1428] Setting key 6747f63ecd3619754891cf93046ed84a
I20260810 06:24:10.369874 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:41917
--builtin_ntp_servers=127.31.204.212:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 06:24:10.372265 31257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:10.364935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:47263 (local address 127.31.204.254:41917)
0810 06:24:10.365089 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:10.365094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:10.366005 (+ 911us) server_negotiation.cc:407] Connection header received
0810 06:24:10.367241 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:10.367244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:10.367288 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:10.367354 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:10.368376 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.369032 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.370497 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.370738 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.370817 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:10.371834 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:10.371838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:10.371840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:10.371853 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:10.371872 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:10.371879 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:10.371935 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:10.372071 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 06:24:10.372131 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 06:24:10.372965 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:10.365293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:41917 (local address 127.31.204.193:47263)
0810 06:24:10.365876 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:10.365911 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:10.367064 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:10.367521 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:10.367531 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:10.367634 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:10.368230 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:10.368248 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.369153 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.369156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:10.370378 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:10.370389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.370874 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.370877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:10.371046 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:10.371563 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:24:10.371565 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:10.371617 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:24:10.371695 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:10.371945 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:10.371954 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:10.371960 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 06:24:10.372064 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 06:24:10.372313 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260810 06:24:10.374150 31504 heartbeater.cc:347] Connected to a master server at 127.31.204.254:41917
I20260810 06:24:10.374346 31504 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:10.374773 31504 heartbeater.cc:511] Master 127.31.204.254:41917 requested a full tablet report, sending...
I20260810 06:24:10.376034 31180 ts_manager.cc:194] Registered new tserver with Master: 9827dc4662304358af1205edfb1c3984 (127.31.204.193:40265)
I20260810 06:24:10.377081 31180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.204.193:47263
W20260810 06:24:10.473805 31507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:10.474040 31507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:10.474078 31507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:10.477999 31507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:10.478061 31507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:10.478114 31507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:24:10.482833 31507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:10.483918 31507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:10.485447 31507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:10.494117 31516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:10.494129 31517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:10.494320 31507 server_base.cc:1034] running on GCE node
W20260810 06:24:10.494342 31519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:10.494871 31507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:10.495417 31507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:10.496567 31507 hybrid_clock.cc:648] HybridClock initialized: now 1786343050496547 us; error 31 us; skew 500 ppm
I20260810 06:24:10.498281 31507 webserver.cc:460] Webserver started at http://127.31.204.194:34541/ using document root <none> and password file <none>
I20260810 06:24:10.498786 31507 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:10.498848 31507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:10.499037 31507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:10.500906 31507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "ac54ad1a839340a58b6f6bdce3698c02"
format_stamp: "Formatted at 2026-08-10 06:24:10 on dist-test-slave-q9h9"
server_key: "6427aa18a58f09c16f657a2590bbb27b"
server_key_iv: "e0c688221dbaaf3e88c00951df9833bd"
server_key_version: "encryptionkey@0"
I20260810 06:24:10.501421 31507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "ac54ad1a839340a58b6f6bdce3698c02"
format_stamp: "Formatted at 2026-08-10 06:24:10 on dist-test-slave-q9h9"
server_key: "6427aa18a58f09c16f657a2590bbb27b"
server_key_iv: "e0c688221dbaaf3e88c00951df9833bd"
server_key_version: "encryptionkey@0"
I20260810 06:24:10.504781 31507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:24:10.506959 31527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:10.508116 31507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:24:10.508222 31507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "ac54ad1a839340a58b6f6bdce3698c02"
format_stamp: "Formatted at 2026-08-10 06:24:10 on dist-test-slave-q9h9"
server_key: "6427aa18a58f09c16f657a2590bbb27b"
server_key_iv: "e0c688221dbaaf3e88c00951df9833bd"
server_key_version: "encryptionkey@0"
I20260810 06:24:10.508311 31507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:10.528576 31507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:10.529192 31507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:10.529356 31507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:10.529908 31507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:10.530818 31507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:10.530867 31507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:10.530917 31507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:10.530946 31507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:10.539197 31507 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:36117
I20260810 06:24:10.539235 31751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:36117 every 8 connection(s)
I20260810 06:24:10.540077 31507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:24:10.546464 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 31507
I20260810 06:24:10.546566 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:24:10.546839 32563 external_mini_cluster.cc:1428] Setting key 4e0d80328fa523eb454f500fba919851
I20260810 06:24:10.548830 31257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:10.541702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:47369 (local address 127.31.204.254:41917)
0810 06:24:10.541830 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:10.541833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:10.542869 (+ 1036us) server_negotiation.cc:407] Connection header received
0810 06:24:10.544222 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:10.544226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:10.544286 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:10.544361 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:10.545228 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.545753 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.546971 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.547277 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.547380 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:10.548377 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:10.548381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:10.548384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:10.548397 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:10.548417 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:10.548424 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:10.548481 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:10.548619 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 06:24:10.548678 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:24:10.548877 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:41917
--builtin_ntp_servers=127.31.204.212:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 06:24:10.549640 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:10.542072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:41917 (local address 127.31.204.194:47369)
0810 06:24:10.542719 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:10.542758 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:10.544011 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:10.544535 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:10.544542 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:10.544631 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:10.545080 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:10.545093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.545873 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.545876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:10.546820 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:10.546831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.547439 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.547444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:10.547601 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:10.548128 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:24:10.548129 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:10.548167 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:24:10.548236 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:10.548494 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:10.548503 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:10.548509 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 06:24:10.548610 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 06:24:10.548866 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":124,"threads_started":1}
I20260810 06:24:10.550661 31753 heartbeater.cc:347] Connected to a master server at 127.31.204.254:41917
I20260810 06:24:10.550879 31753 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:10.551331 31753 heartbeater.cc:511] Master 127.31.204.254:41917 requested a full tablet report, sending...
I20260810 06:24:10.552253 31180 ts_manager.cc:194] Registered new tserver with Master: ac54ad1a839340a58b6f6bdce3698c02 (127.31.204.194:36117)
I20260810 06:24:10.552776 31180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.204.194:47369
W20260810 06:24:10.667461 31759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:10.667721 31759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:10.667779 31759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:10.671695 31759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:10.671746 31759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:10.671825 31759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:24:10.676429 31759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:10.677404 31759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:10.678782 31759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:10.688066 31768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:10.688164 31767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:10.688253 31759 server_base.cc:1034] running on GCE node
W20260810 06:24:10.688066 31771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:10.688746 31759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:10.689291 31759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:10.690438 31759 hybrid_clock.cc:648] HybridClock initialized: now 1786343050690422 us; error 30 us; skew 500 ppm
I20260810 06:24:10.692241 31759 webserver.cc:460] Webserver started at http://127.31.204.195:44641/ using document root <none> and password file <none>
I20260810 06:24:10.692734 31759 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:10.692798 31759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:10.692958 31759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:10.694890 31759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "ff4292749c2a42cea89011068061d591"
format_stamp: "Formatted at 2026-08-10 06:24:10 on dist-test-slave-q9h9"
server_key: "57c26b734bd01fcb6e496fe94445b92b"
server_key_iv: "7b32b7eb46f097182efc6980e45e6ab9"
server_key_version: "encryptionkey@0"
I20260810 06:24:10.695369 31759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "ff4292749c2a42cea89011068061d591"
format_stamp: "Formatted at 2026-08-10 06:24:10 on dist-test-slave-q9h9"
server_key: "57c26b734bd01fcb6e496fe94445b92b"
server_key_iv: "7b32b7eb46f097182efc6980e45e6ab9"
server_key_version: "encryptionkey@0"
I20260810 06:24:10.698866 31759 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:24:10.701026 31784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:10.702167 31759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:24:10.702257 31759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "ff4292749c2a42cea89011068061d591"
format_stamp: "Formatted at 2026-08-10 06:24:10 on dist-test-slave-q9h9"
server_key: "57c26b734bd01fcb6e496fe94445b92b"
server_key_iv: "7b32b7eb46f097182efc6980e45e6ab9"
server_key_version: "encryptionkey@0"
I20260810 06:24:10.702348 31759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:10.731773 31759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:10.732427 31759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:10.732595 31759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:10.733155 31759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:10.734123 31759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:10.734172 31759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:10.734225 31759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:10.734252 31759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:10.742928 31759 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:40339
I20260810 06:24:10.742970 31991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:40339 every 8 connection(s)
I20260810 06:24:10.743755 31759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:24:10.746879 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 31759
I20260810 06:24:10.746974 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:24:10.747231 32563 external_mini_cluster.cc:1428] Setting key 7de8415961fa35e1446345c36e6f9301
I20260810 06:24:10.752743 31257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:10.745446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:34731 (local address 127.31.204.254:41917)
0810 06:24:10.745604 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:10.745608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:10.746629 (+ 1021us) server_negotiation.cc:407] Connection header received
0810 06:24:10.748159 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:10.748161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:10.748212 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:10.748303 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:10.749218 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.749752 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.750912 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.751142 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.751243 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:10.752238 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:10.752241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:10.752244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:10.752256 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:10.752277 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:10.752287 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:10.752353 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:10.752506 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 06:24:10.752581 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 06:24:10.753479 31999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:10.745807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:41917 (local address 127.31.204.195:34731)
0810 06:24:10.746473 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:10.746514 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:10.747946 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:10.748458 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:10.748467 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:10.748575 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:10.749048 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:10.749078 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.749883 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.749886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:10.750777 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:10.750788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.751290 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.751294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:10.751455 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:10.751976 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:24:10.751978 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:10.752016 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:24:10.752088 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:10.752363 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:10.752372 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:10.752379 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 06:24:10.752484 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 06:24:10.752746 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":124,"threads_started":1}
I20260810 06:24:10.754485 31996 heartbeater.cc:347] Connected to a master server at 127.31.204.254:41917
I20260810 06:24:10.754685 31996 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:10.755143 31996 heartbeater.cc:511] Master 127.31.204.254:41917 requested a full tablet report, sending...
I20260810 06:24:10.756093 31180 ts_manager.cc:194] Registered new tserver with Master: ff4292749c2a42cea89011068061d591 (127.31.204.195:40339)
I20260810 06:24:10.756639 31180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.204.195:34731
I20260810 06:24:10.760756 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:24:10.765810 31257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:10.762601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37662 (local address 127.31.204.254:41917)
0810 06:24:10.762740 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:10.762745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:10.762891 (+ 146us) server_negotiation.cc:407] Connection header received
0810 06:24:10.762975 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:10.762977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:10.763016 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:10.763082 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:10.763280 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.763759 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.764476 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.764678 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.764746 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:10.765431 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:10.765434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:10.765436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:10.765446 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:10.765461 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:10.765468 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:10.765563 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:10.765632 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 06:24:10.765666 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 06:24:10.768386 31180 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:24:10.771303 31180 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:24:10.784787 32020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:10.779858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:40339 (local address 127.0.0.1:43104)
0810 06:24:10.780427 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:10.780440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:10.780526 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:10.781155 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:10.781158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:10.781176 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:10.781258 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:10.781264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.782193 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.782196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:10.783418 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:10.783429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.784279 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.784282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:10.784384 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:10.784419 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 06:24:10.784528 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 06:24:10.784645 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":87,"threads_started":1}
I20260810 06:24:10.784803 32023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:10.780399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43104 (local address 127.31.204.195:40339)
0810 06:24:10.780755 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:10.780767 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:10.780790 (+ 23us) server_negotiation.cc:407] Connection header received
0810 06:24:10.780861 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:10.780866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:10.781026 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:10.781155 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:10.781357 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.782087 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.783557 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.784170 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.784280 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:10.784454 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:10.784555 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 06:24:10.784646 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":100,"threads_started":1}
I20260810 06:24:10.785004 32017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:10.779186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:36117 (local address 127.0.0.1:54958)
0810 06:24:10.779496 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:10.779539 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:10.779651 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:10.780671 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:10.780677 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:10.780701 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:10.780900 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:10.780909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.782610 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.782613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:10.783569 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:10.783580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.784614 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.784618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:10.784719 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:10.784745 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 06:24:10.784790 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 06:24:10.784856 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":6,"thread_start_us":65,"threads_started":1}
I20260810 06:24:10.785005 32018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:10.779427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:40265 (local address 127.0.0.1:56730)
0810 06:24:10.779801 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:10.779815 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:10.779904 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:10.780740 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:10.780742 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:10.780756 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:10.780912 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:10.780918 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.782574 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.782578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:10.783710 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:10.783719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.784699 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.784702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:10.784775 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:10.784795 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 06:24:10.784832 (+ 37us) client_negotiation.cc:240] Negotiation successful
0810 06:24:10.784881 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":40,"thread_start_us":98,"threads_started":1}
I20260810 06:24:10.785575 32022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:10.779944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.31.204.193:40265)
0810 06:24:10.780346 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:10.780359 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:10.780380 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:24:10.780446 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:10.780450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:10.780610 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:10.781084 (+ 474us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:10.781248 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.782452 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.783832 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.784587 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.785097 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:10.785280 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:10.785331 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 06:24:10.785418 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":97,"threads_started":1}
I20260810 06:24:10.785575 32019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:10.779419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54958 (local address 127.31.204.194:36117)
0810 06:24:10.780160 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:10.780172 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:10.780197 (+ 25us) server_negotiation.cc:407] Connection header received
0810 06:24:10.780285 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:10.780290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:10.780495 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:10.781083 (+ 588us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:10.781232 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.782452 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.783720 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:10.784472 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:10.785130 (+ 658us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:10.785280 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:10.785331 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 06:24:10.785415 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":108,"threads_started":1}
I20260810 06:24:10.787961 31873 tablet_service.cc:1467] Processing CreateTablet for tablet 62852eca3f494187a84eee8fe5951c6f (DEFAULT_TABLE table=test-table [id=05751486cf554e8c9b5f9cad8e7669c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:24:10.788061 31619 tablet_service.cc:1467] Processing CreateTablet for tablet 62852eca3f494187a84eee8fe5951c6f (DEFAULT_TABLE table=test-table [id=05751486cf554e8c9b5f9cad8e7669c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:24:10.788905 31619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62852eca3f494187a84eee8fe5951c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:10.788905 31873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62852eca3f494187a84eee8fe5951c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:10.791942 31397 tablet_service.cc:1467] Processing CreateTablet for tablet 62852eca3f494187a84eee8fe5951c6f (DEFAULT_TABLE table=test-table [id=05751486cf554e8c9b5f9cad8e7669c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:24:10.792970 31397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62852eca3f494187a84eee8fe5951c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:10.795538 32024 tablet_bootstrap.cc:492] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591: Bootstrap starting.
I20260810 06:24:10.796415 32026 tablet_bootstrap.cc:492] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02: Bootstrap starting.
I20260810 06:24:10.797647 32024 tablet_bootstrap.cc:654] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:10.798429 32024 log.cc:824] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:10.798686 32026 tablet_bootstrap.cc:654] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:10.799516 32026 log.cc:824] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:10.800132 32034 tablet_bootstrap.cc:492] T 62852eca3f494187a84eee8fe5951c6f P 9827dc4662304358af1205edfb1c3984: Bootstrap starting.
I20260810 06:24:10.800357 32024 tablet_bootstrap.cc:492] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591: No bootstrap required, opened a new log
I20260810 06:24:10.800539 32024 ts_tablet_manager.cc:1397] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 06:24:10.801294 32026 tablet_bootstrap.cc:492] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02: No bootstrap required, opened a new log
I20260810 06:24:10.801438 32026 ts_tablet_manager.cc:1397] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:24:10.802294 32034 tablet_bootstrap.cc:654] T 62852eca3f494187a84eee8fe5951c6f P 9827dc4662304358af1205edfb1c3984: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:10.803138 32034 log.cc:824] T 62852eca3f494187a84eee8fe5951c6f P 9827dc4662304358af1205edfb1c3984: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:10.804526 32024 raft_consensus.cc:348] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac54ad1a839340a58b6f6bdce3698c02" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36117 } } peers { permanent_uuid: "ff4292749c2a42cea89011068061d591" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40339 } } peers { permanent_uuid: "9827dc4662304358af1205edfb1c3984" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40265 } }
I20260810 06:24:10.804780 32024 raft_consensus.cc:374] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:10.804868 32024 raft_consensus.cc:729] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff4292749c2a42cea89011068061d591, State: Initialized, Role: FOLLOWER
I20260810 06:24:10.805006 32026 raft_consensus.cc:348] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac54ad1a839340a58b6f6bdce3698c02" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36117 } } peers { permanent_uuid: "ff4292749c2a42cea89011068061d591" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40339 } } peers { permanent_uuid: "9827dc4662304358af1205edfb1c3984" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40265 } }
I20260810 06:24:10.805160 32034 tablet_bootstrap.cc:492] T 62852eca3f494187a84eee8fe5951c6f P 9827dc4662304358af1205edfb1c3984: No bootstrap required, opened a new log
I20260810 06:24:10.805230 32026 raft_consensus.cc:374] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:10.805280 32026 raft_consensus.cc:729] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac54ad1a839340a58b6f6bdce3698c02, State: Initialized, Role: FOLLOWER
I20260810 06:24:10.805446 32034 ts_tablet_manager.cc:1397] T 62852eca3f494187a84eee8fe5951c6f P 9827dc4662304358af1205edfb1c3984: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260810 06:24:10.805423 32024 consensus_queue.cc:260] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac54ad1a839340a58b6f6bdce3698c02" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36117 } } peers { permanent_uuid: "ff4292749c2a42cea89011068061d591" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40339 } } peers { permanent_uuid: "9827dc4662304358af1205edfb1c3984" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40265 } }
I20260810 06:24:10.805727 32026 consensus_queue.cc:260] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac54ad1a839340a58b6f6bdce3698c02" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36117 } } peers { permanent_uuid: "ff4292749c2a42cea89011068061d591" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40339 } } peers { permanent_uuid: "9827dc4662304358af1205edfb1c3984" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40265 } }
I20260810 06:24:10.806195 31996 heartbeater.cc:503] Master 127.31.204.254:41917 was elected leader, sending a full tablet report...
I20260810 06:24:10.806581 32024 ts_tablet_manager.cc:1428] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260810 06:24:10.807677 32026 ts_tablet_manager.cc:1428] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 06:24:10.808013 31753 heartbeater.cc:503] Master 127.31.204.254:41917 was elected leader, sending a full tablet report...
I20260810 06:24:10.809177 32034 raft_consensus.cc:348] T 62852eca3f494187a84eee8fe5951c6f P 9827dc4662304358af1205edfb1c3984 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac54ad1a839340a58b6f6bdce3698c02" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36117 } } peers { permanent_uuid: "ff4292749c2a42cea89011068061d591" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40339 } } peers { permanent_uuid: "9827dc4662304358af1205edfb1c3984" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40265 } }
I20260810 06:24:10.809480 32034 raft_consensus.cc:374] T 62852eca3f494187a84eee8fe5951c6f P 9827dc4662304358af1205edfb1c3984 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:10.809552 32034 raft_consensus.cc:729] T 62852eca3f494187a84eee8fe5951c6f P 9827dc4662304358af1205edfb1c3984 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9827dc4662304358af1205edfb1c3984, State: Initialized, Role: FOLLOWER
I20260810 06:24:10.809960 32034 consensus_queue.cc:260] T 62852eca3f494187a84eee8fe5951c6f P 9827dc4662304358af1205edfb1c3984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac54ad1a839340a58b6f6bdce3698c02" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36117 } } peers { permanent_uuid: "ff4292749c2a42cea89011068061d591" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40339 } } peers { permanent_uuid: "9827dc4662304358af1205edfb1c3984" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40265 } }
I20260810 06:24:10.810652 31504 heartbeater.cc:503] Master 127.31.204.254:41917 was elected leader, sending a full tablet report...
I20260810 06:24:10.810909 32034 ts_tablet_manager.cc:1428] T 62852eca3f494187a84eee8fe5951c6f P 9827dc4662304358af1205edfb1c3984: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260810 06:24:10.864463 31505 tablet.cc:2369] T 62852eca3f494187a84eee8fe5951c6f P 9827dc4662304358af1205edfb1c3984: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:24:10.994817 31997 tablet.cc:2369] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:24:11.041389 31754 tablet.cc:2369] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:24:11.193338 32040 raft_consensus.cc:482] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:24:11.193573 32040 raft_consensus.cc:504] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac54ad1a839340a58b6f6bdce3698c02" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36117 } } peers { permanent_uuid: "ff4292749c2a42cea89011068061d591" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40339 } } peers { permanent_uuid: "9827dc4662304358af1205edfb1c3984" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40265 } }
I20260810 06:24:11.194751 32040 leader_election.cc:288] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff4292749c2a42cea89011068061d591 (127.31.204.195:40339), 9827dc4662304358af1205edfb1c3984 (127.31.204.193:40265)
I20260810 06:24:11.198544 32022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:11.195194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:50005 (local address 127.31.204.193:40265)
0810 06:24:11.195325 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:11.195329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:11.195498 (+ 169us) server_negotiation.cc:407] Connection header received
0810 06:24:11.195611 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:11.195614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:11.195660 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:11.195723 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:11.195976 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:11.196520 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:11.197640 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:11.198058 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:11.198173 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:11.198226 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:11.198367 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 06:24:11.198435 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 06:24:11.198561 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:11.195021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:40265 (local address 127.31.204.194:50005)
0810 06:24:11.195402 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:11.195413 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:11.195514 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:11.195741 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:11.195744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:11.195759 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:11.195872 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:11.195877 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:11.196621 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:11.196632 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:11.197523 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:11.197533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:11.198160 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:11.198164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:11.198256 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:11.198279 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 06:24:11.198344 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 06:24:11.198407 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":32,"thread_start_us":65,"threads_started":1}
I20260810 06:24:11.198609 32023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:11.195102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:45193 (local address 127.31.204.195:40339)
0810 06:24:11.195250 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:11.195254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:11.195426 (+ 172us) server_negotiation.cc:407] Connection header received
0810 06:24:11.195565 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:11.195568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:11.195611 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:11.195672 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:11.195929 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:11.196549 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:11.197689 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:11.198135 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:11.198240 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:11.198296 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:11.198440 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 06:24:11.198506 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 06:24:11.198628 32044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:11.195005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:40339 (local address 127.31.204.194:45193)
0810 06:24:11.195337 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:11.195350 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:11.195463 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:11.195690 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:11.195693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:11.195708 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:11.195831 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:11.195836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:11.196657 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:11.196660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:11.197570 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:11.197580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:11.198247 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:11.198250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:11.198333 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:11.198354 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 06:24:11.198420 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 06:24:11.198484 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":11,"thread_start_us":132,"threads_started":1}
I20260810 06:24:11.199268 31429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62852eca3f494187a84eee8fe5951c6f" candidate_uuid: "ac54ad1a839340a58b6f6bdce3698c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9827dc4662304358af1205edfb1c3984" is_pre_election: true
I20260810 06:24:11.199545 31429 raft_consensus.cc:2454] T 62852eca3f494187a84eee8fe5951c6f P 9827dc4662304358af1205edfb1c3984 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac54ad1a839340a58b6f6bdce3698c02 in term 0.
I20260810 06:24:11.199571 31905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62852eca3f494187a84eee8fe5951c6f" candidate_uuid: "ac54ad1a839340a58b6f6bdce3698c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff4292749c2a42cea89011068061d591" is_pre_election: true
I20260810 06:24:11.199836 31905 raft_consensus.cc:2454] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac54ad1a839340a58b6f6bdce3698c02 in term 0.
I20260810 06:24:11.199982 31536 leader_election.cc:302] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9827dc4662304358af1205edfb1c3984, ac54ad1a839340a58b6f6bdce3698c02; no voters:
I20260810 06:24:11.200234 32040 raft_consensus.cc:2781] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:24:11.200320 32040 raft_consensus.cc:482] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:24:11.200366 32040 raft_consensus.cc:3037] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:11.201341 32040 raft_consensus.cc:504] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac54ad1a839340a58b6f6bdce3698c02" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36117 } } peers { permanent_uuid: "ff4292749c2a42cea89011068061d591" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40339 } } peers { permanent_uuid: "9827dc4662304358af1205edfb1c3984" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40265 } }
I20260810 06:24:11.201637 32040 leader_election.cc:288] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [CANDIDATE]: Term 1 election: Requested vote from peers ff4292749c2a42cea89011068061d591 (127.31.204.195:40339), 9827dc4662304358af1205edfb1c3984 (127.31.204.193:40265)
I20260810 06:24:11.201951 31429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62852eca3f494187a84eee8fe5951c6f" candidate_uuid: "ac54ad1a839340a58b6f6bdce3698c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9827dc4662304358af1205edfb1c3984"
I20260810 06:24:11.201949 31905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62852eca3f494187a84eee8fe5951c6f" candidate_uuid: "ac54ad1a839340a58b6f6bdce3698c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff4292749c2a42cea89011068061d591"
I20260810 06:24:11.202093 31905 raft_consensus.cc:3037] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:11.202093 31429 raft_consensus.cc:3037] T 62852eca3f494187a84eee8fe5951c6f P 9827dc4662304358af1205edfb1c3984 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:11.203271 31905 raft_consensus.cc:2454] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac54ad1a839340a58b6f6bdce3698c02 in term 1.
I20260810 06:24:11.203280 31429 raft_consensus.cc:2454] T 62852eca3f494187a84eee8fe5951c6f P 9827dc4662304358af1205edfb1c3984 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac54ad1a839340a58b6f6bdce3698c02 in term 1.
I20260810 06:24:11.203553 31536 leader_election.cc:302] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9827dc4662304358af1205edfb1c3984, ac54ad1a839340a58b6f6bdce3698c02, ff4292749c2a42cea89011068061d591; no voters:
I20260810 06:24:11.203714 32040 raft_consensus.cc:2781] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:11.203931 32040 raft_consensus.cc:686] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [term 1 LEADER]: Becoming Leader. State: Replica: ac54ad1a839340a58b6f6bdce3698c02, State: Running, Role: LEADER
I20260810 06:24:11.204247 32040 consensus_queue.cc:237] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac54ad1a839340a58b6f6bdce3698c02" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36117 } } peers { permanent_uuid: "ff4292749c2a42cea89011068061d591" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40339 } } peers { permanent_uuid: "9827dc4662304358af1205edfb1c3984" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40265 } }
I20260810 06:24:11.207208 31180 catalog_manager.cc:5495] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac54ad1a839340a58b6f6bdce3698c02 (127.31.204.194). New cstate: current_term: 1 leader_uuid: "ac54ad1a839340a58b6f6bdce3698c02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac54ad1a839340a58b6f6bdce3698c02" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff4292749c2a42cea89011068061d591" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9827dc4662304358af1205edfb1c3984" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40265 } health_report { overall_health: UNKNOWN } } }
I20260810 06:24:11.235448 32019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:11.231987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54962 (local address 127.31.204.194:36117)
0810 06:24:11.232100 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:11.232104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:11.232116 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:24:11.232252 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:11.232254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:11.232313 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:11.232403 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:24:11.232629 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:11.233098 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:11.233986 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:11.234218 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:11.234308 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:11.234670 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:24:11.235043 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:24:11.235172 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:11.235300 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 06:24:11.235335 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:24:11.240870 31905 raft_consensus.cc:1261] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591 [term 1 FOLLOWER]: Refusing update from remote peer ac54ad1a839340a58b6f6bdce3698c02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:24:11.240876 31429 raft_consensus.cc:1261] T 62852eca3f494187a84eee8fe5951c6f P 9827dc4662304358af1205edfb1c3984 [term 1 FOLLOWER]: Refusing update from remote peer ac54ad1a839340a58b6f6bdce3698c02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:24:11.241500 32051 consensus_queue.cc:1035] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff4292749c2a42cea89011068061d591" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 40339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:24:11.241658 32040 consensus_queue.cc:1035] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9827dc4662304358af1205edfb1c3984" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:24:11.251389 32055 mvcc.cc:205] Tried to move back new op lower bound from 7316861137876283392 to 7316861137735008256. Current Snapshot: MvccSnapshot[applied={T|T < 7316861137876283392 or (T in {7316861137876283392})}]
I20260810 06:24:11.256098 31180 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:24:11.256256 31180 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:24:11.258513 31180 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cc05db9f7a546f1a2114246b7f1c894: Sending DeleteTablet for 3 replicas of tablet 62852eca3f494187a84eee8fe5951c6f
I20260810 06:24:11.259240 31397 tablet_service.cc:1514] Processing DeleteTablet for tablet 62852eca3f494187a84eee8fe5951c6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:24:11 UTC) from {username='slave'} at 127.0.0.1:56730
I20260810 06:24:11.259321 31619 tablet_service.cc:1514] Processing DeleteTablet for tablet 62852eca3f494187a84eee8fe5951c6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:24:11 UTC) from {username='slave'} at 127.0.0.1:54958
I20260810 06:24:11.259800 32077 tablet_replica.cc:330] T 62852eca3f494187a84eee8fe5951c6f P 9827dc4662304358af1205edfb1c3984: stopping tablet replica
I20260810 06:24:11.259810 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 31296
I20260810 06:24:11.259788 31873 tablet_service.cc:1514] Processing DeleteTablet for tablet 62852eca3f494187a84eee8fe5951c6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:24:11 UTC) from {username='slave'} at 127.0.0.1:43104
I20260810 06:24:11.260140 32077 raft_consensus.cc:2229] T 62852eca3f494187a84eee8fe5951c6f P 9827dc4662304358af1205edfb1c3984 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:24:11.260236 32079 tablet_replica.cc:330] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591: stopping tablet replica
I20260810 06:24:11.260464 32079 raft_consensus.cc:2229] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:24:11.260711 32079 raft_consensus.cc:2258] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:24:11.261025 32076 tablet_replica.cc:330] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02: stopping tablet replica
I20260810 06:24:11.261300 32076 raft_consensus.cc:2229] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:24:11.261610 32076 raft_consensus.cc:2258] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:24:11.263756 32076 ts_tablet_manager.cc:1905] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:24:11.263756 32079 ts_tablet_manager.cc:1905] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:24:11.266613 32079 ts_tablet_manager.cc:1918] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:24:11.266784 32079 log.cc:1196] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/62852eca3f494187a84eee8fe5951c6f
I20260810 06:24:11.266779 32076 ts_tablet_manager.cc:1918] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:24:11.266863 32076 log.cc:1196] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/62852eca3f494187a84eee8fe5951c6f
I20260810 06:24:11.267242 32076 ts_tablet_manager.cc:1939] T 62852eca3f494187a84eee8fe5951c6f P ac54ad1a839340a58b6f6bdce3698c02: Deleting consensus metadata
I20260810 06:24:11.267242 32079 ts_tablet_manager.cc:1939] T 62852eca3f494187a84eee8fe5951c6f P ff4292749c2a42cea89011068061d591: Deleting consensus metadata
I20260810 06:24:11.268395 31151 catalog_manager.cc:4841] TS ac54ad1a839340a58b6f6bdce3698c02 (127.31.204.194:36117): tablet 62852eca3f494187a84eee8fe5951c6f (table test-table [id=05751486cf554e8c9b5f9cad8e7669c7]) successfully deleted
I20260810 06:24:11.268604 31151 catalog_manager.cc:4841] TS ff4292749c2a42cea89011068061d591 (127.31.204.195:40339): tablet 62852eca3f494187a84eee8fe5951c6f (table test-table [id=05751486cf554e8c9b5f9cad8e7669c7]) successfully deleted
W20260810 06:24:11.275485 31157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.204.193:40265 (error 108)
I20260810 06:24:11.276248 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 31507
I20260810 06:24:11.276523 32020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:11.276175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:40265 (local address 127.0.0.1:56744)
0810 06:24:11.276362 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:11.276438 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:40265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260810 06:24:11.278920 31157 catalog_manager.cc:4584] TS 9827dc4662304358af1205edfb1c3984 (127.31.204.193:40265): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62852eca3f494187a84eee8fe5951c6f: Network error: Client connection negotiation failed: client connection to 127.31.204.193:40265: connect: Connection refused (error 111)
I20260810 06:24:11.289228 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 31759
I20260810 06:24:11.299863 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 31123
2026-08-10T06:24:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1381 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:11 dist-test-slave-q9h9 krb5kdc[32090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:11 dist-test-slave-q9h9 krb5kdc[32090](info): set up 2 sockets
Aug 10 06:24:11 dist-test-slave-q9h9 krb5kdc[32090](info): commencing operation
krb5kdc: starting...
W20260810 06:24:12.450505 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:12 dist-test-slave-q9h9 krb5kdc[32090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:24:12.561151 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:35747
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:36501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:35747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:12.670651 32107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:12.670897 32107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:12.670934 32107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:12.674737 32107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:12.674778 32107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:12.674796 32107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:12.674826 32107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:12.674844 32107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:12.679783 32107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:36501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:35747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:35747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:12.680665 32107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:12.681581 32107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:12.689448 32120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:12.689457 32118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:12.689570 32107 server_base.cc:1034] running on GCE node
W20260810 06:24:12.689450 32123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:12.690331 32107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:12.691335 32107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:12.692490 32107 hybrid_clock.cc:648] HybridClock initialized: now 1786343052692479 us; error 28 us; skew 500 ppm
Aug 10 06:24:12 dist-test-slave-q9h9 krb5kdc[32090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343052, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:12.695083 32107 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:24:12.695900 32107 webserver.cc:460] Webserver started at http://127.31.204.254:41489/ using document root <none> and password file <none>
I20260810 06:24:12.696382 32107 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:12.696436 32107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:12.696591 32107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:12.698524 32107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "ac5419009a3747998fbe44e05ad99b33"
format_stamp: "Formatted at 2026-08-10 06:24:12 on dist-test-slave-q9h9"
server_key: "e048342291fdfef8466d51e1caf4e72d"
server_key_iv: "6902733de1ab5e4580659ec8aed95f02"
server_key_version: "encryptionkey@0"
I20260810 06:24:12.699013 32107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "ac5419009a3747998fbe44e05ad99b33"
format_stamp: "Formatted at 2026-08-10 06:24:12 on dist-test-slave-q9h9"
server_key: "e048342291fdfef8466d51e1caf4e72d"
server_key_iv: "6902733de1ab5e4580659ec8aed95f02"
server_key_version: "encryptionkey@0"
I20260810 06:24:12.702404 32107 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:24:12.704545 32136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:12.705649 32107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:12.705763 32107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "ac5419009a3747998fbe44e05ad99b33"
format_stamp: "Formatted at 2026-08-10 06:24:12 on dist-test-slave-q9h9"
server_key: "e048342291fdfef8466d51e1caf4e72d"
server_key_iv: "6902733de1ab5e4580659ec8aed95f02"
server_key_version: "encryptionkey@0"
I20260810 06:24:12.705847 32107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:12.720291 32107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:12.722708 32107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:12.722856 32107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:12.729120 32107 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:35747
I20260810 06:24:12.729161 32224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:35747 every 8 connection(s)
I20260810 06:24:12.729995 32107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:12.732553 32225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:12.736227 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 32107
I20260810 06:24:12.736356 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:12.736613 32563 external_mini_cluster.cc:1428] Setting key ca621e08bbd7d4d26c477bcbe0decd07
I20260810 06:24:12.738061 32225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33: Bootstrap starting.
I20260810 06:24:12.740423 32225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:12.741173 32225 log.cc:824] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:12.743134 32225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33: No bootstrap required, opened a new log
Aug 10 06:24:12 dist-test-slave-q9h9 krb5kdc[32090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:24:12.747041 32225 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5419009a3747998fbe44e05ad99b33" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35747 } }
I20260810 06:24:12.747256 32225 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:12.747314 32225 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac5419009a3747998fbe44e05ad99b33, State: Initialized, Role: FOLLOWER
I20260810 06:24:12.747721 32225 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5419009a3747998fbe44e05ad99b33" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35747 } }
I20260810 06:24:12.747848 32225 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:12.747913 32225 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:12.748009 32225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:12.748821 32225 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5419009a3747998fbe44e05ad99b33" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35747 } }
I20260810 06:24:12.749130 32225 leader_election.cc:302] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac5419009a3747998fbe44e05ad99b33; no voters:
I20260810 06:24:12.749396 32225 leader_election.cc:288] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:12.749505 32232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:12.749706 32232 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33 [term 1 LEADER]: Becoming Leader. State: Replica: ac5419009a3747998fbe44e05ad99b33, State: Running, Role: LEADER
I20260810 06:24:12.750129 32230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:12.737823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56268 (local address 127.31.204.254:35747)
0810 06:24:12.738199 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:12.738214 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:12.738248 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:24:12.738852 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:12.738868 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:12.739162 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:12.739460 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:12.740324 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:12.741374 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:12.742457 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:12.742904 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:12.743016 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:12.745574 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:12.745578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:12.745590 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:12.745615 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:12.747184 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:12.747613 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:12.747618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:12.747622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:12.747726 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:12.748001 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:12.748004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:12.748006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:12.748387 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:12.748572 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:12.748851 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 06:24:12.749149 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":115,"threads_started":1}
I20260810 06:24:12.750083 32232 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5419009a3747998fbe44e05ad99b33" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35747 } }
I20260810 06:24:12.750581 32225 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:12.751230 32232 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac5419009a3747998fbe44e05ad99b33. Latest consensus state: current_term: 1 leader_uuid: "ac5419009a3747998fbe44e05ad99b33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5419009a3747998fbe44e05ad99b33" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35747 } } }
I20260810 06:24:12.751354 32232 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:12.752002 32233 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac5419009a3747998fbe44e05ad99b33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5419009a3747998fbe44e05ad99b33" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35747 } } }
I20260810 06:24:12.752079 32233 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:12.752341 32245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:12.755316 32245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:12.760424 32245 catalog_manager.cc:1347] Generated new cluster ID: 0169de58384b4196ad1208e5aa0d6070
I20260810 06:24:12.760484 32245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:12.769591 32245 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:12.770450 32245 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:12.777616 32245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33: Generated new TSK 0
I20260810 06:24:12.778235 32245 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.204.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
Entry for principal kudu/127.31.204.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab.
I20260810 06:24:12.808233 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35747
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:36501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:12.935750 32262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:12.936017 32262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:12.936062 32262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:12.940133 32262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:12.940186 32262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:12.940258 32262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:24:12.945533 32262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:36501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.193.keytab
--principal=kudu/127.31.204.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:12.946586 32262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:12.947628 32262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:12.956601 32272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:12.956619 32268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:12.956684 32269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:12.956807 32262 server_base.cc:1034] running on GCE node
I20260810 06:24:12.957244 32262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:12.957821 32262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:12.958962 32262 hybrid_clock.cc:648] HybridClock initialized: now 1786343052958941 us; error 31 us; skew 500 ppm
Aug 10 06:24:12 dist-test-slave-q9h9 krb5kdc[32090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343052, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:12.961469 32262 init.cc:377] Logged in from keytab as kudu/127.31.204.193@KRBTEST.COM (short username kudu)
I20260810 06:24:12.962231 32262 webserver.cc:460] Webserver started at http://127.31.204.193:41505/ using document root <none> and password file <none>
I20260810 06:24:12.962695 32262 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:12.962751 32262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:12.962910 32262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:12.964825 32262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "86d98bbb773c4bbeae7c142acec88850"
format_stamp: "Formatted at 2026-08-10 06:24:12 on dist-test-slave-q9h9"
server_key: "ce077adad9f2227a2fb114922c62c2e7"
server_key_iv: "4ea4d26892837cdecd2347591b03c533"
server_key_version: "encryptionkey@0"
I20260810 06:24:12.965323 32262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "86d98bbb773c4bbeae7c142acec88850"
format_stamp: "Formatted at 2026-08-10 06:24:12 on dist-test-slave-q9h9"
server_key: "ce077adad9f2227a2fb114922c62c2e7"
server_key_iv: "4ea4d26892837cdecd2347591b03c533"
server_key_version: "encryptionkey@0"
I20260810 06:24:12.968636 32262 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 06:24:12.970757 32286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:12.971832 32262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:12.971920 32262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "86d98bbb773c4bbeae7c142acec88850"
format_stamp: "Formatted at 2026-08-10 06:24:12 on dist-test-slave-q9h9"
server_key: "ce077adad9f2227a2fb114922c62c2e7"
server_key_iv: "4ea4d26892837cdecd2347591b03c533"
server_key_version: "encryptionkey@0"
I20260810 06:24:12.971995 32262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:12.994376 32262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:12.996793 32262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:12.996923 32262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:12.997362 32262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:12.998116 32262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:12.998162 32262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:12.998196 32262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:12.998216 32262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:13.006218 32262 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:40753
I20260810 06:24:13.006278 32460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:40753 every 8 connection(s)
I20260810 06:24:13.006969 32262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:24:13.008265 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 32262
I20260810 06:24:13.008399 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:24:13.008704 32563 external_mini_cluster.cc:1428] Setting key e42d50f0f3d80850059b3eb80648e8cd
Aug 10 06:24:13 dist-test-slave-q9h9 krb5kdc[32090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343052, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.193@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.204.194@KRBTEST.COM; defaulting to no policy
I20260810 06:24:13.019944 32230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.008604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:56339 (local address 127.31.204.254:35747)
0810 06:24:13.008744 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:13.008749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:13.009486 (+ 737us) server_negotiation.cc:407] Connection header received
0810 06:24:13.011047 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:13.011051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:13.011107 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:13.011192 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:13.012203 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.012725 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.013789 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.014038 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.014134 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:13.016628 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:13.016631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:13.016633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:13.016655 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:13.017933 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:13.018489 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:13.018491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:13.018493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:13.018589 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:13.019017 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:13.019020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:13.019021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:13.019217 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:13.019303 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:13.019731 (+ 428us) server_negotiation.cc:299] Negotiation successful
0810 06:24:13.019830 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:24:13.020663 32463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.008888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:35747 (local address 127.31.204.193:56339)
0810 06:24:13.009336 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:13.009375 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:13.010827 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:13.011339 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:13.011345 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:13.011647 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:13.012058 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:13.012070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.012841 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.012843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:13.013677 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:13.013686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.014174 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.014178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:13.014324 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:13.014816 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:13.014832 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:13.016462 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:13.018074 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:13.018080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:13.018094 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:13.018398 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:13.018669 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:13.018673 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:13.018675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:13.018852 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:13.019300 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:13.019305 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:13.019542 (+ 237us) client_negotiation.cc:769] Sending connection context
0810 06:24:13.019681 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 06:24:13.019907 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":135,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.194@KRBTEST.COM" created.
I20260810 06:24:13.021746 32461 heartbeater.cc:347] Connected to a master server at 127.31.204.254:35747
I20260810 06:24:13.021974 32461 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:13.022493 32461 heartbeater.cc:511] Master 127.31.204.254:35747 requested a full tablet report, sending...
I20260810 06:24:13.023932 32161 ts_manager.cc:194] Registered new tserver with Master: 86d98bbb773c4bbeae7c142acec88850 (127.31.204.193:40753)
I20260810 06:24:13.025033 32161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.193@KRBTEST.COM'} at 127.31.204.193:56339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
Entry for principal kudu/127.31.204.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab.
I20260810 06:24:13.034078 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35747
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:36501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:13.136852 32467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:13.137061 32467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:13.137127 32467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:13.140985 32467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:13.141036 32467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:13.141114 32467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:24:13.145955 32467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:36501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.194.keytab
--principal=kudu/127.31.204.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:13.146770 32467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:13.147601 32467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:13.156378 32474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:13.156352 32479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:13.156383 32476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:13.156487 32467 server_base.cc:1034] running on GCE node
I20260810 06:24:13.156962 32467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:13.157546 32467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:13.158704 32467 hybrid_clock.cc:648] HybridClock initialized: now 1786343053158685 us; error 30 us; skew 500 ppm
Aug 10 06:24:13 dist-test-slave-q9h9 krb5kdc[32090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343053, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:13.161195 32467 init.cc:377] Logged in from keytab as kudu/127.31.204.194@KRBTEST.COM (short username kudu)
I20260810 06:24:13.161968 32467 webserver.cc:460] Webserver started at http://127.31.204.194:38651/ using document root <none> and password file <none>
I20260810 06:24:13.162510 32467 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:13.162575 32467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:13.162753 32467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:13.164623 32467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "736668786d9b49f4b74cf1697d88a60f"
format_stamp: "Formatted at 2026-08-10 06:24:13 on dist-test-slave-q9h9"
server_key: "840526c4e9fec7472f95fae3a47dfe23"
server_key_iv: "73613a007cf7989d679086b8621868b3"
server_key_version: "encryptionkey@0"
I20260810 06:24:13.165212 32467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "736668786d9b49f4b74cf1697d88a60f"
format_stamp: "Formatted at 2026-08-10 06:24:13 on dist-test-slave-q9h9"
server_key: "840526c4e9fec7472f95fae3a47dfe23"
server_key_iv: "73613a007cf7989d679086b8621868b3"
server_key_version: "encryptionkey@0"
I20260810 06:24:13.168721 32467 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:24:13.170882 32495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:13.171986 32467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:13.172089 32467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "736668786d9b49f4b74cf1697d88a60f"
format_stamp: "Formatted at 2026-08-10 06:24:13 on dist-test-slave-q9h9"
server_key: "840526c4e9fec7472f95fae3a47dfe23"
server_key_iv: "73613a007cf7989d679086b8621868b3"
server_key_version: "encryptionkey@0"
I20260810 06:24:13.172190 32467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:13.188561 32467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:13.191123 32467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:13.191295 32467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:13.191812 32467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:13.192569 32467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:13.192615 32467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:13.192665 32467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:13.192694 32467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:13.206874 32467 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:44033
I20260810 06:24:13.206910 32703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:44033 every 8 connection(s)
I20260810 06:24:13.207746 32467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:24:13.210278 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 32467
I20260810 06:24:13.210409 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:24:13.210700 32563 external_mini_cluster.cc:1428] Setting key ae2f0ceec3d4ed6d05bfd0c98e57d409
Aug 10 06:24:13 dist-test-slave-q9h9 krb5kdc[32090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343053, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.194@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.204.195@KRBTEST.COM; defaulting to no policy
I20260810 06:24:13.220584 32230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.209366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:44995 (local address 127.31.204.254:35747)
0810 06:24:13.209514 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:13.209519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:13.210361 (+ 842us) server_negotiation.cc:407] Connection header received
0810 06:24:13.211646 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:13.211649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:13.211690 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:13.211745 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:13.213075 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.213587 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.214905 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.215127 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.215225 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:13.217419 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:13.217422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:13.217424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:13.217446 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:13.218727 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:13.219186 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:13.219190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:13.219191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:13.219266 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:13.219562 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:13.219565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:13.219567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:13.219734 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:13.219819 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:13.220303 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 06:24:13.220425 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 06:24:13.221333 32706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.209754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:35747 (local address 127.31.204.194:44995)
0810 06:24:13.210226 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:13.210268 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:13.211452 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:13.211900 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:13.211907 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:13.212327 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:13.212932 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:13.212952 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.213708 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.213712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:13.214775 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:13.214785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.215271 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.215275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:13.215430 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:13.215919 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:13.215935 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:13.217266 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:13.218835 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:13.218841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:13.218854 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:13.219089 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:13.219347 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:13.219350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:13.219352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:13.219477 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:13.219832 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:13.219838 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:13.220086 (+ 248us) client_negotiation.cc:769] Sending connection context
0810 06:24:13.220243 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 06:24:13.220536 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.195@KRBTEST.COM" created.
I20260810 06:24:13.222425 32704 heartbeater.cc:347] Connected to a master server at 127.31.204.254:35747
I20260810 06:24:13.222663 32704 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:13.223198 32704 heartbeater.cc:511] Master 127.31.204.254:35747 requested a full tablet report, sending...
I20260810 06:24:13.224252 32161 ts_manager.cc:194] Registered new tserver with Master: 736668786d9b49f4b74cf1697d88a60f (127.31.204.194:44033)
I20260810 06:24:13.224869 32161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.194@KRBTEST.COM'} at 127.31.204.194:44995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
Entry for principal kudu/127.31.204.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab.
I20260810 06:24:13.234645 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35747
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:36501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:13.341370 32709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:13.341629 32709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:13.341693 32709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:13.345588 32709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:13.345644 32709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:13.345721 32709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:24:13.350760 32709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:36501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.195.keytab
--principal=kudu/127.31.204.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:35747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:13.351735 32709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:13.352705 32709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:24:13.362088 32709 server_base.cc:1034] running on GCE node
W20260810 06:24:13.361996 32721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:13.361977 32720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:13.362006 32724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:13.362699 32709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:13.363301 32709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:13.364447 32709 hybrid_clock.cc:648] HybridClock initialized: now 1786343053364422 us; error 39 us; skew 500 ppm
Aug 10 06:24:13 dist-test-slave-q9h9 krb5kdc[32090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343053, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:13.367081 32709 init.cc:377] Logged in from keytab as kudu/127.31.204.195@KRBTEST.COM (short username kudu)
I20260810 06:24:13.367954 32709 webserver.cc:460] Webserver started at http://127.31.204.195:35001/ using document root <none> and password file <none>
I20260810 06:24:13.368491 32709 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:13.368559 32709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:13.368770 32709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:13.370811 32709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "1acb7726677348618e1659ee46750c6a"
format_stamp: "Formatted at 2026-08-10 06:24:13 on dist-test-slave-q9h9"
server_key: "ecb42dfb7066630d1f961f1e0316ea09"
server_key_iv: "1b298b43dac7de8af8ec57b129fc4684"
server_key_version: "encryptionkey@0"
I20260810 06:24:13.371340 32709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "1acb7726677348618e1659ee46750c6a"
format_stamp: "Formatted at 2026-08-10 06:24:13 on dist-test-slave-q9h9"
server_key: "ecb42dfb7066630d1f961f1e0316ea09"
server_key_iv: "1b298b43dac7de8af8ec57b129fc4684"
server_key_version: "encryptionkey@0"
I20260810 06:24:13.375020 32709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:24:13.377316 32734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:13.378487 32709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:24:13.378598 32709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "1acb7726677348618e1659ee46750c6a"
format_stamp: "Formatted at 2026-08-10 06:24:13 on dist-test-slave-q9h9"
server_key: "ecb42dfb7066630d1f961f1e0316ea09"
server_key_iv: "1b298b43dac7de8af8ec57b129fc4684"
server_key_version: "encryptionkey@0"
I20260810 06:24:13.378685 32709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:13.392051 32709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:13.394618 32709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:13.394786 32709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:13.395332 32709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:13.396107 32709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:13.396153 32709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:13.396202 32709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:13.396241 32709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:13.404778 32709 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:46787
I20260810 06:24:13.404798 457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:46787 every 8 connection(s)
I20260810 06:24:13.405571 32709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:24:13.410043 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 32709
I20260810 06:24:13.410159 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:24:13.410419 32563 external_mini_cluster.cc:1428] Setting key c69e07d15a4c492735bc3534293cc023
Aug 10 06:24:13 dist-test-slave-q9h9 krb5kdc[32090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343053, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.195@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:24:13.418800 32230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.407230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:50937 (local address 127.31.204.254:35747)
0810 06:24:13.407350 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:13.407353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:13.408156 (+ 803us) server_negotiation.cc:407] Connection header received
0810 06:24:13.409508 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:13.409512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:13.409574 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:13.409645 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:13.410957 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.411659 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.412882 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.413100 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.413188 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:13.415509 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:13.415512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:13.415515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:13.415539 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:13.416852 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:13.417323 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:13.417326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:13.417327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:13.417404 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:13.417742 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:13.417746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:13.417747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:13.417921 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:13.418008 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:13.418521 (+ 513us) server_negotiation.cc:299] Negotiation successful
0810 06:24:13.418642 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:24:13.419548 460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.407537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:35747 (local address 127.31.204.195:50937)
0810 06:24:13.407992 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:13.408034 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:13.409281 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:13.409826 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:13.409833 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:13.410273 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:13.410818 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:13.410840 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.411786 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.411788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:13.412750 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:13.412769 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.413222 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.413224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:13.413361 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:13.413838 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:13.413853 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:13.415353 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:13.416961 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:13.416967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:13.416981 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:13.417230 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:13.417488 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:13.417491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:13.417493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:13.417646 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:13.418044 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:13.418051 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:13.418298 (+ 247us) client_negotiation.cc:769] Sending connection context
0810 06:24:13.418460 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 06:24:13.418773 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":126,"threads_started":1}
I20260810 06:24:13.420585 458 heartbeater.cc:347] Connected to a master server at 127.31.204.254:35747
I20260810 06:24:13.420801 458 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:13.421226 458 heartbeater.cc:511] Master 127.31.204.254:35747 requested a full tablet report, sending...
I20260810 06:24:13.422171 32161 ts_manager.cc:194] Registered new tserver with Master: 1acb7726677348618e1659ee46750c6a (127.31.204.195:46787)
I20260810 06:24:13.422739 32161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.204.195@KRBTEST.COM'} at 127.31.204.195:50937
I20260810 06:24:13.423986 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:24:13.432569 32230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.426010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56276 (local address 127.31.204.254:35747)
0810 06:24:13.426153 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:13.426158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:13.426335 (+ 177us) server_negotiation.cc:407] Connection header received
0810 06:24:13.426438 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:13.426441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:13.426483 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:13.426544 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:13.427192 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.427688 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.428484 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.428701 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.428785 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:13.429966 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:13.429970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:13.429973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:13.429997 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:13.431047 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:13.431436 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:13.431439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:13.431441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:13.431519 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:13.431768 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:13.431771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:13.431773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:13.431978 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:13.432096 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:13.432326 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 06:24:13.432429 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 06:24:13.435094 32160 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:24:13.437940 32160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:24:13.451953 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.446191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60426 (local address 127.31.204.195:46787)
0810 06:24:13.446643 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:13.446654 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:13.446876 (+ 222us) server_negotiation.cc:407] Connection header received
0810 06:24:13.447093 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:13.447098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:13.447316 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:13.447441 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:13.447917 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.449091 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.450547 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.451220 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.451347 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:13.451481 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:13.451656 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 06:24:13.451759 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":143,"threads_started":1}
I20260810 06:24:13.452095 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.446822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.31.204.193:40753)
0810 06:24:13.447309 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:13.447321 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:13.447343 (+ 22us) server_negotiation.cc:407] Connection header received
0810 06:24:13.447417 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:13.447422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:13.447630 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:13.448067 (+ 437us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:13.448218 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.449259 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.450495 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.451372 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.451570 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:13.451704 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:13.451859 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 06:24:13.451956 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":125,"threads_started":1}
I20260810 06:24:13.452400 476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.445951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:46787 (local address 127.0.0.1:60426)
0810 06:24:13.446783 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:13.446795 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:13.446949 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:13.447628 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:13.447634 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:13.447660 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:13.447809 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:13.447816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.449232 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.449235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:13.450415 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:13.450424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.451335 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.451338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:13.451449 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:13.451487 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 06:24:13.452143 (+ 656us) client_negotiation.cc:240] Negotiation successful
0810 06:24:13.452210 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":54,"thread_start_us":114,"threads_started":1}
I20260810 06:24:13.452425 474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.445891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:44033 (local address 127.0.0.1:37430)
0810 06:24:13.446288 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:13.446342 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:13.446467 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:13.447694 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:13.447697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:13.447710 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:13.447812 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:13.447817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.448913 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.448916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:13.450219 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:13.450230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.451508 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.451512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:13.451641 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:13.451685 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 06:24:13.451741 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 06:24:13.451904 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":25,"thread_start_us":86,"threads_started":1}
I20260810 06:24:13.452427 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.446828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.31.204.194:44033)
0810 06:24:13.447266 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:13.447277 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:13.447299 (+ 22us) server_negotiation.cc:407] Connection header received
0810 06:24:13.447371 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:13.447376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:13.447543 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:13.447953 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:13.448079 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.448783 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.450357 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.451372 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.452025 (+ 653us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:13.452154 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:13.452207 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 06:24:13.452294 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":126,"threads_started":1}
I20260810 06:24:13.452965 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.446664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:40753 (local address 127.0.0.1:45678)
0810 06:24:13.446958 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:13.446972 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:13.447062 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:13.447786 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:13.447789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:13.447805 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:13.447920 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:13.447927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.449380 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.449384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:13.450357 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:13.450367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.451508 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.451512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:13.451689 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:13.451711 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:24:13.451790 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 06:24:13.451906 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":102,"thread_start_us":91,"threads_started":1}
I20260810 06:24:13.455538 32361 tablet_service.cc:1467] Processing CreateTablet for tablet 36e9988fe1674e87b9170909b3cf3ac3 (DEFAULT_TABLE table=test-table [id=abeee7a4da2a4329a22c0482b5edc1f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:24:13.455528 346 tablet_service.cc:1467] Processing CreateTablet for tablet 36e9988fe1674e87b9170909b3cf3ac3 (DEFAULT_TABLE table=test-table [id=abeee7a4da2a4329a22c0482b5edc1f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:24:13.455526 32582 tablet_service.cc:1467] Processing CreateTablet for tablet 36e9988fe1674e87b9170909b3cf3ac3 (DEFAULT_TABLE table=test-table [id=abeee7a4da2a4329a22c0482b5edc1f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:24:13.456485 32361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36e9988fe1674e87b9170909b3cf3ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:13.456506 32582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36e9988fe1674e87b9170909b3cf3ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:13.456506 346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36e9988fe1674e87b9170909b3cf3ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:13.462867 488 tablet_bootstrap.cc:492] T 36e9988fe1674e87b9170909b3cf3ac3 P 86d98bbb773c4bbeae7c142acec88850: Bootstrap starting.
I20260810 06:24:13.463374 489 tablet_bootstrap.cc:492] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a: Bootstrap starting.
I20260810 06:24:13.464166 491 tablet_bootstrap.cc:492] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f: Bootstrap starting.
I20260810 06:24:13.465329 488 tablet_bootstrap.cc:654] T 36e9988fe1674e87b9170909b3cf3ac3 P 86d98bbb773c4bbeae7c142acec88850: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:13.465461 489 tablet_bootstrap.cc:654] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:13.465991 491 tablet_bootstrap.cc:654] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:13.466725 489 log.cc:824] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:13.466725 488 log.cc:824] T 36e9988fe1674e87b9170909b3cf3ac3 P 86d98bbb773c4bbeae7c142acec88850: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:13.467033 491 log.cc:824] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:13.468819 491 tablet_bootstrap.cc:492] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f: No bootstrap required, opened a new log
I20260810 06:24:13.468828 489 tablet_bootstrap.cc:492] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a: No bootstrap required, opened a new log
I20260810 06:24:13.468876 488 tablet_bootstrap.cc:492] T 36e9988fe1674e87b9170909b3cf3ac3 P 86d98bbb773c4bbeae7c142acec88850: No bootstrap required, opened a new log
I20260810 06:24:13.469009 491 ts_tablet_manager.cc:1397] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:24:13.469009 489 ts_tablet_manager.cc:1397] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:24:13.469058 488 ts_tablet_manager.cc:1397] T 36e9988fe1674e87b9170909b3cf3ac3 P 86d98bbb773c4bbeae7c142acec88850: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 06:24:13.472046 488 raft_consensus.cc:348] T 36e9988fe1674e87b9170909b3cf3ac3 P 86d98bbb773c4bbeae7c142acec88850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736668786d9b49f4b74cf1697d88a60f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44033 } } peers { permanent_uuid: "1acb7726677348618e1659ee46750c6a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 46787 } } peers { permanent_uuid: "86d98bbb773c4bbeae7c142acec88850" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40753 } }
I20260810 06:24:13.472271 488 raft_consensus.cc:374] T 36e9988fe1674e87b9170909b3cf3ac3 P 86d98bbb773c4bbeae7c142acec88850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:13.472344 488 raft_consensus.cc:729] T 36e9988fe1674e87b9170909b3cf3ac3 P 86d98bbb773c4bbeae7c142acec88850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86d98bbb773c4bbeae7c142acec88850, State: Initialized, Role: FOLLOWER
I20260810 06:24:13.472791 488 consensus_queue.cc:260] T 36e9988fe1674e87b9170909b3cf3ac3 P 86d98bbb773c4bbeae7c142acec88850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736668786d9b49f4b74cf1697d88a60f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44033 } } peers { permanent_uuid: "1acb7726677348618e1659ee46750c6a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 46787 } } peers { permanent_uuid: "86d98bbb773c4bbeae7c142acec88850" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40753 } }
I20260810 06:24:13.472736 489 raft_consensus.cc:348] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736668786d9b49f4b74cf1697d88a60f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44033 } } peers { permanent_uuid: "1acb7726677348618e1659ee46750c6a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 46787 } } peers { permanent_uuid: "86d98bbb773c4bbeae7c142acec88850" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40753 } }
I20260810 06:24:13.472736 491 raft_consensus.cc:348] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736668786d9b49f4b74cf1697d88a60f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44033 } } peers { permanent_uuid: "1acb7726677348618e1659ee46750c6a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 46787 } } peers { permanent_uuid: "86d98bbb773c4bbeae7c142acec88850" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40753 } }
I20260810 06:24:13.472985 491 raft_consensus.cc:374] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:13.473013 489 raft_consensus.cc:374] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:13.473066 489 raft_consensus.cc:729] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1acb7726677348618e1659ee46750c6a, State: Initialized, Role: FOLLOWER
I20260810 06:24:13.473098 491 raft_consensus.cc:729] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 736668786d9b49f4b74cf1697d88a60f, State: Initialized, Role: FOLLOWER
I20260810 06:24:13.473450 32461 heartbeater.cc:503] Master 127.31.204.254:35747 was elected leader, sending a full tablet report...
I20260810 06:24:13.473539 489 consensus_queue.cc:260] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736668786d9b49f4b74cf1697d88a60f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44033 } } peers { permanent_uuid: "1acb7726677348618e1659ee46750c6a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 46787 } } peers { permanent_uuid: "86d98bbb773c4bbeae7c142acec88850" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40753 } }
I20260810 06:24:13.473539 491 consensus_queue.cc:260] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736668786d9b49f4b74cf1697d88a60f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44033 } } peers { permanent_uuid: "1acb7726677348618e1659ee46750c6a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 46787 } } peers { permanent_uuid: "86d98bbb773c4bbeae7c142acec88850" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40753 } }
I20260810 06:24:13.473785 488 ts_tablet_manager.cc:1428] T 36e9988fe1674e87b9170909b3cf3ac3 P 86d98bbb773c4bbeae7c142acec88850: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260810 06:24:13.474503 489 ts_tablet_manager.cc:1428] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 06:24:13.474745 491 ts_tablet_manager.cc:1428] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 06:24:13.474774 458 heartbeater.cc:503] Master 127.31.204.254:35747 was elected leader, sending a full tablet report...
I20260810 06:24:13.474982 505 raft_consensus.cc:482] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:24:13.475092 505 raft_consensus.cc:504] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736668786d9b49f4b74cf1697d88a60f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44033 } } peers { permanent_uuid: "1acb7726677348618e1659ee46750c6a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 46787 } } peers { permanent_uuid: "86d98bbb773c4bbeae7c142acec88850" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40753 } }
I20260810 06:24:13.476050 32704 heartbeater.cc:503] Master 127.31.204.254:35747 was elected leader, sending a full tablet report...
I20260810 06:24:13.477938 505 leader_election.cc:288] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1acb7726677348618e1659ee46750c6a (127.31.204.195:46787), 86d98bbb773c4bbeae7c142acec88850 (127.31.204.193:40753)
I20260810 06:24:13.481393 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.477025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:41767 (local address 127.31.204.195:46787)
0810 06:24:13.477126 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:13.477131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:13.477143 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:24:13.477182 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:13.477185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:13.477226 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:13.477314 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:13.478234 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.478921 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.480265 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.480869 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.480955 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:13.480993 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:13.481249 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 06:24:13.481306 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 06:24:13.481410 32706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.476689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:46787 (local address 127.31.204.194:41767)
0810 06:24:13.476969 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:13.476984 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:13.477067 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:13.477717 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:13.477720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:13.477738 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:13.477851 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:13.477859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.479045 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.479048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:13.480154 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:13.480163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.480990 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.480994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:13.481103 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:13.481129 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 06:24:13.481214 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 06:24:13.481278 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":23}
I20260810 06:24:13.481961 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.477673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:40753 (local address 127.31.204.194:52183)
0810 06:24:13.478222 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:13.478235 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:13.478375 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:13.478590 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:13.478593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:13.478608 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:13.478722 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:13.478728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.479638 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.479640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:13.480698 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:13.480708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.481564 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.481567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:13.481674 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:13.481698 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 06:24:13.481763 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 06:24:13.481825 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260810 06:24:13.481945 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.478045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:52183 (local address 127.31.204.193:40753)
0810 06:24:13.478164 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:13.478168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:13.478311 (+ 143us) server_negotiation.cc:407] Connection header received
0810 06:24:13.478469 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:13.478472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:13.478512 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:13.478570 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:13.478848 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.479522 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.480838 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.481414 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.481533 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:13.481580 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:13.481779 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 06:24:13.481848 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:24:13.482059 381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e9988fe1674e87b9170909b3cf3ac3" candidate_uuid: "736668786d9b49f4b74cf1697d88a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1acb7726677348618e1659ee46750c6a" is_pre_election: true
I20260810 06:24:13.482329 381 raft_consensus.cc:2454] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 736668786d9b49f4b74cf1697d88a60f in term 0.
I20260810 06:24:13.482576 32389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e9988fe1674e87b9170909b3cf3ac3" candidate_uuid: "736668786d9b49f4b74cf1697d88a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d98bbb773c4bbeae7c142acec88850" is_pre_election: true
I20260810 06:24:13.482769 32497 leader_election.cc:302] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1acb7726677348618e1659ee46750c6a, 736668786d9b49f4b74cf1697d88a60f; no voters:
I20260810 06:24:13.482838 32389 raft_consensus.cc:2454] T 36e9988fe1674e87b9170909b3cf3ac3 P 86d98bbb773c4bbeae7c142acec88850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 736668786d9b49f4b74cf1697d88a60f in term 0.
I20260810 06:24:13.483018 505 raft_consensus.cc:2781] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:24:13.483093 505 raft_consensus.cc:482] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:24:13.483166 505 raft_consensus.cc:3037] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:13.484143 505 raft_consensus.cc:504] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736668786d9b49f4b74cf1697d88a60f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44033 } } peers { permanent_uuid: "1acb7726677348618e1659ee46750c6a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 46787 } } peers { permanent_uuid: "86d98bbb773c4bbeae7c142acec88850" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40753 } }
I20260810 06:24:13.484431 505 leader_election.cc:288] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [CANDIDATE]: Term 1 election: Requested vote from peers 1acb7726677348618e1659ee46750c6a (127.31.204.195:46787), 86d98bbb773c4bbeae7c142acec88850 (127.31.204.193:40753)
I20260810 06:24:13.484690 381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e9988fe1674e87b9170909b3cf3ac3" candidate_uuid: "736668786d9b49f4b74cf1697d88a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1acb7726677348618e1659ee46750c6a"
I20260810 06:24:13.484757 32389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e9988fe1674e87b9170909b3cf3ac3" candidate_uuid: "736668786d9b49f4b74cf1697d88a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d98bbb773c4bbeae7c142acec88850"
I20260810 06:24:13.484809 381 raft_consensus.cc:3037] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:13.484855 32389 raft_consensus.cc:3037] T 36e9988fe1674e87b9170909b3cf3ac3 P 86d98bbb773c4bbeae7c142acec88850 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:13.486033 381 raft_consensus.cc:2454] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 736668786d9b49f4b74cf1697d88a60f in term 1.
I20260810 06:24:13.486033 32389 raft_consensus.cc:2454] T 36e9988fe1674e87b9170909b3cf3ac3 P 86d98bbb773c4bbeae7c142acec88850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 736668786d9b49f4b74cf1697d88a60f in term 1.
I20260810 06:24:13.486299 32497 leader_election.cc:302] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 736668786d9b49f4b74cf1697d88a60f, 86d98bbb773c4bbeae7c142acec88850; no voters:
I20260810 06:24:13.486436 505 raft_consensus.cc:2781] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:13.486538 505 raft_consensus.cc:686] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [term 1 LEADER]: Becoming Leader. State: Replica: 736668786d9b49f4b74cf1697d88a60f, State: Running, Role: LEADER
I20260810 06:24:13.486832 505 consensus_queue.cc:237] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736668786d9b49f4b74cf1697d88a60f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44033 } } peers { permanent_uuid: "1acb7726677348618e1659ee46750c6a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 46787 } } peers { permanent_uuid: "86d98bbb773c4bbeae7c142acec88850" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40753 } }
I20260810 06:24:13.489410 32160 catalog_manager.cc:5495] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f reported cstate change: term changed from 0 to 1, leader changed from <none> to 736668786d9b49f4b74cf1697d88a60f (127.31.204.194). New cstate: current_term: 1 leader_uuid: "736668786d9b49f4b74cf1697d88a60f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736668786d9b49f4b74cf1697d88a60f" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1acb7726677348618e1659ee46750c6a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 46787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86d98bbb773c4bbeae7c142acec88850" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40753 } health_report { overall_health: UNKNOWN } } }
I20260810 06:24:13.502888 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.499951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37442 (local address 127.31.204.194:44033)
0810 06:24:13.500069 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:13.500072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:13.500085 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:24:13.500147 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:13.500149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:13.500197 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:13.500280 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:24:13.500457 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.500931 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.501680 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.501879 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.501950 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:13.502216 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:24:13.502588 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:24:13.502685 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:13.502716 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 06:24:13.502762 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 06:24:13.507992 32462 tablet.cc:2369] T 36e9988fe1674e87b9170909b3cf3ac3 P 86d98bbb773c4bbeae7c142acec88850: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:24:13.508397 32389 raft_consensus.cc:1261] T 36e9988fe1674e87b9170909b3cf3ac3 P 86d98bbb773c4bbeae7c142acec88850 [term 1 FOLLOWER]: Refusing update from remote peer 736668786d9b49f4b74cf1697d88a60f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:24:13.508417 381 raft_consensus.cc:1261] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a [term 1 FOLLOWER]: Refusing update from remote peer 736668786d9b49f4b74cf1697d88a60f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:24:13.509055 500 consensus_queue.cc:1035] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "86d98bbb773c4bbeae7c142acec88850" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:24:13.509347 505 consensus_queue.cc:1035] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1acb7726677348618e1659ee46750c6a" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 46787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:24:13.515632 511 mvcc.cc:205] Tried to move back new op lower bound from 7316861147163398144 to 7316861147084267520. Current Snapshot: MvccSnapshot[applied={T|T < 7316861147163398144}]
I20260810 06:24:13.517146 512 mvcc.cc:205] Tried to move back new op lower bound from 7316861147163398144 to 7316861147084267520. Current Snapshot: MvccSnapshot[applied={T|T < 7316861147163398144}]
I20260810 06:24:13.518275 514 mvcc.cc:205] Tried to move back new op lower bound from 7316861147163398144 to 7316861147084267520. Current Snapshot: MvccSnapshot[applied={T|T < 7316861147163398144}]
I20260810 06:24:13.524034 32161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:24:13.524228 32161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:24:13.526684 32161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac5419009a3747998fbe44e05ad99b33: Sending DeleteTablet for 3 replicas of tablet 36e9988fe1674e87b9170909b3cf3ac3
I20260810 06:24:13.527417 32582 tablet_service.cc:1514] Processing DeleteTablet for tablet 36e9988fe1674e87b9170909b3cf3ac3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:24:13 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:37430
I20260810 06:24:13.527484 346 tablet_service.cc:1514] Processing DeleteTablet for tablet 36e9988fe1674e87b9170909b3cf3ac3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:24:13 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:60426
I20260810 06:24:13.527946 527 tablet_replica.cc:330] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a: stopping tablet replica
I20260810 06:24:13.527925 32361 tablet_service.cc:1514] Processing DeleteTablet for tablet 36e9988fe1674e87b9170909b3cf3ac3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:24:13 UTC) from {username='kudu', principal='kudu/127.31.204.254@KRBTEST.COM'} at 127.0.0.1:45678
I20260810 06:24:13.528206 528 tablet_replica.cc:330] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f: stopping tablet replica
I20260810 06:24:13.528334 535 tablet_replica.cc:330] T 36e9988fe1674e87b9170909b3cf3ac3 P 86d98bbb773c4bbeae7c142acec88850: stopping tablet replica
I20260810 06:24:13.528646 527 raft_consensus.cc:2229] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:24:13.528699 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 32262
I20260810 06:24:13.531625 528 raft_consensus.cc:2229] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:24:13.532089 528 raft_consensus.cc:2258] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:24:13.532449 527 raft_consensus.cc:2258] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:24:13.535742 527 ts_tablet_manager.cc:1905] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:24:13.538475 527 ts_tablet_manager.cc:1918] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:24:13.538570 527 log.cc:1196] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/36e9988fe1674e87b9170909b3cf3ac3
I20260810 06:24:13.538915 527 ts_tablet_manager.cc:1939] T 36e9988fe1674e87b9170909b3cf3ac3 P 1acb7726677348618e1659ee46750c6a: Deleting consensus metadata
I20260810 06:24:13.539273 528 ts_tablet_manager.cc:1905] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:24:13.540102 32137 catalog_manager.cc:4841] TS 1acb7726677348618e1659ee46750c6a (127.31.204.195:46787): tablet 36e9988fe1674e87b9170909b3cf3ac3 (table test-table [id=abeee7a4da2a4329a22c0482b5edc1f7]) successfully deleted
W20260810 06:24:13.541604 32137 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.204.193:40753 (error 108)
I20260810 06:24:13.542248 528 ts_tablet_manager.cc:1918] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:24:13.542348 528 log.cc:1196] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/36e9988fe1674e87b9170909b3cf3ac3
I20260810 06:24:13.542672 528 ts_tablet_manager.cc:1939] T 36e9988fe1674e87b9170909b3cf3ac3 P 736668786d9b49f4b74cf1697d88a60f: Deleting consensus metadata
I20260810 06:24:13.542770 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.542458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:40753 (local address 127.0.0.1:45688)
0810 06:24:13.542601 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:13.542672 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 06:24:13.542857 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 32467
W20260810 06:24:13.566278 32137 catalog_manager.cc:4584] TS 86d98bbb773c4bbeae7c142acec88850 (127.31.204.193:40753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 36e9988fe1674e87b9170909b3cf3ac3: Network error: Client connection negotiation failed: client connection to 127.31.204.193:40753: connect: Connection refused (error 111)
I20260810 06:24:13.571643 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 32709
I20260810 06:24:13.572103 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.571780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:44033 (local address 127.0.0.1:37456)
0810 06:24:13.571963 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:13.572013 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.194:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260810 06:24:13.582334 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 32107
2026-08-10T06:24:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2286 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T06:24:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:13Z Disabled control of system clock
I20260810 06:24:13.618312 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:41045
--webserver_interface=127.31.204.254
--webserver_port=0
--builtin_ntp_servers=127.31.204.212:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:41045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 06:24:13.727053 544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:13.727281 544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:13.727315 544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:13.731204 544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:13.731249 544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:13.731266 544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:13.731295 544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:13.731323 544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:13.735868 544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:41045
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:41045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:13.736814 544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:13.738154 544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:13.746273 556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:13.746340 544 server_base.cc:1034] running on GCE node
W20260810 06:24:13.746449 554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:13.746536 553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:13.747061 544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:13.748051 544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:13.749222 544 hybrid_clock.cc:648] HybridClock initialized: now 1786343053749201 us; error 27 us; skew 500 ppm
I20260810 06:24:13.750913 544 webserver.cc:460] Webserver started at http://127.31.204.254:36027/ using document root <none> and password file <none>
I20260810 06:24:13.751430 544 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:13.751497 544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:13.751688 544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:13.753604 544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "8d3dc3a5255b4ff5ba80ab357d286c7a"
format_stamp: "Formatted at 2026-08-10 06:24:13 on dist-test-slave-q9h9"
server_key: "9becade461aa48c358e905e6c485cc26"
server_key_iv: "d5692652e9905d9f86747be541ede7e9"
server_key_version: "encryptionkey@0"
I20260810 06:24:13.754187 544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "8d3dc3a5255b4ff5ba80ab357d286c7a"
format_stamp: "Formatted at 2026-08-10 06:24:13 on dist-test-slave-q9h9"
server_key: "9becade461aa48c358e905e6c485cc26"
server_key_iv: "d5692652e9905d9f86747be541ede7e9"
server_key_version: "encryptionkey@0"
I20260810 06:24:13.757550 544 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:24:13.759820 562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:13.760905 544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:13.761016 544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "8d3dc3a5255b4ff5ba80ab357d286c7a"
format_stamp: "Formatted at 2026-08-10 06:24:13 on dist-test-slave-q9h9"
server_key: "9becade461aa48c358e905e6c485cc26"
server_key_iv: "d5692652e9905d9f86747be541ede7e9"
server_key_version: "encryptionkey@0"
I20260810 06:24:13.761102 544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:13.784192 544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:13.784814 544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:13.784976 544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:13.791808 544 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:41045
I20260810 06:24:13.791826 643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:41045 every 8 connection(s)
I20260810 06:24:13.792728 544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:13.793529 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 544
I20260810 06:24:13.793660 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:13.793985 32563 external_mini_cluster.cc:1428] Setting key b1c687ce4b8062e972c32fcceeafe60c
I20260810 06:24:13.796248 644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:13.801604 644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a: Bootstrap starting.
I20260810 06:24:13.802753 650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:13.795394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.31.204.254:41045)
0810 06:24:13.796012 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:13.796027 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:13.796062 (+ 35us) server_negotiation.cc:407] Connection header received
0810 06:24:13.796726 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:13.796743 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:13.797117 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:13.797430 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:13.797835 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.798815 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.799668 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:13.800119 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:13.800270 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:13.801096 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:13.801100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:13.801112 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:13.801478 (+ 366us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:13.801529 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:13.801549 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:13.801632 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:13.801837 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 06:24:13.802094 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":143,"threads_started":1}
I20260810 06:24:13.803824 644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:13.804679 644 log.cc:824] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:13.806469 644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a: No bootstrap required, opened a new log
I20260810 06:24:13.809978 644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3dc3a5255b4ff5ba80ab357d286c7a" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 41045 } }
I20260810 06:24:13.810210 644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:13.810266 644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3dc3a5255b4ff5ba80ab357d286c7a, State: Initialized, Role: FOLLOWER
I20260810 06:24:13.810652 644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3dc3a5255b4ff5ba80ab357d286c7a" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 41045 } }
I20260810 06:24:13.810776 644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:13.810837 644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:13.810930 644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:13.811892 644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3dc3a5255b4ff5ba80ab357d286c7a" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 41045 } }
I20260810 06:24:13.812183 644 leader_election.cc:302] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d3dc3a5255b4ff5ba80ab357d286c7a; no voters:
I20260810 06:24:13.812415 644 leader_election.cc:288] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:13.812506 653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:13.812716 653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a [term 1 LEADER]: Becoming Leader. State: Replica: 8d3dc3a5255b4ff5ba80ab357d286c7a, State: Running, Role: LEADER
I20260810 06:24:13.813012 653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3dc3a5255b4ff5ba80ab357d286c7a" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 41045 } }
I20260810 06:24:13.813516 644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:13.814505 655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d3dc3a5255b4ff5ba80ab357d286c7a. Latest consensus state: current_term: 1 leader_uuid: "8d3dc3a5255b4ff5ba80ab357d286c7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3dc3a5255b4ff5ba80ab357d286c7a" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 41045 } } }
I20260810 06:24:13.814625 655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:13.815004 665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:13.815512 654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d3dc3a5255b4ff5ba80ab357d286c7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3dc3a5255b4ff5ba80ab357d286c7a" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 41045 } } }
I20260810 06:24:13.815603 654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:13.817580 665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:13.822768 665 catalog_manager.cc:1347] Generated new cluster ID: 260120faf300487d9b0c5dc5c7f2346f
I20260810 06:24:13.822829 665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:13.830303 665 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:13.831394 665 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:13.836465 665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a: Generated new TSK 0
I20260810 06:24:13.837123 665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:13.840845 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.193:0
--local_ip_for_outbound_sockets=127.31.204.193
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:41045
--builtin_ntp_servers=127.31.204.212:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 06:24:13.949666 678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:13.949905 678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:13.949957 678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:13.954082 678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:13.954146 678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:13.954233 678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.193
I20260810 06:24:13.958969 678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.204.193
--webserver_port=0
--tserver_master_addrs=127.31.204.254:41045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.193
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:13.959935 678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:13.961258 678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:13.970548 693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:13.970556 697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:13.970600 678 server_base.cc:1034] running on GCE node
W20260810 06:24:13.970564 695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:13.971101 678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:13.971697 678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:13.972855 678 hybrid_clock.cc:648] HybridClock initialized: now 1786343053972837 us; error 29 us; skew 500 ppm
I20260810 06:24:13.977054 678 webserver.cc:460] Webserver started at http://127.31.204.193:44819/ using document root <none> and password file <none>
I20260810 06:24:13.977578 678 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:13.977650 678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:13.977836 678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:13.979815 678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-0/data/instance:
uuid: "59aba2f3323d4024aa1b5144be1a4893"
format_stamp: "Formatted at 2026-08-10 06:24:13 on dist-test-slave-q9h9"
server_key: "9645ca579580e4ebad38685265f1bb3e"
server_key_iv: "2bf1c5f74ddb8604307f94656d5c4adc"
server_key_version: "encryptionkey@0"
I20260810 06:24:13.980347 678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance:
uuid: "59aba2f3323d4024aa1b5144be1a4893"
format_stamp: "Formatted at 2026-08-10 06:24:13 on dist-test-slave-q9h9"
server_key: "9645ca579580e4ebad38685265f1bb3e"
server_key_iv: "2bf1c5f74ddb8604307f94656d5c4adc"
server_key_version: "encryptionkey@0"
I20260810 06:24:13.983871 678 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:24:13.986053 705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:13.987195 678 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:13.987309 678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-0/wal
uuid: "59aba2f3323d4024aa1b5144be1a4893"
format_stamp: "Formatted at 2026-08-10 06:24:13 on dist-test-slave-q9h9"
server_key: "9645ca579580e4ebad38685265f1bb3e"
server_key_iv: "2bf1c5f74ddb8604307f94656d5c4adc"
server_key_version: "encryptionkey@0"
I20260810 06:24:13.987397 678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:13.997475 678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:13.998070 678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:13.998230 678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:13.998751 678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:13.999598 678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:13.999643 678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:13.999693 678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:13.999722 678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:14.007812 678 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.193:38887
I20260810 06:24:14.007853 883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.193:38887 every 8 connection(s)
I20260810 06:24:14.008622 678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-0/data/info.pb
I20260810 06:24:14.017314 650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:14.010291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.193:39979 (local address 127.31.204.254:41045)
0810 06:24:14.010429 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:14.010433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:14.011423 (+ 990us) server_negotiation.cc:407] Connection header received
0810 06:24:14.012747 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:14.012751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:14.012806 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:14.012878 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:14.013728 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.014312 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.015512 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.015731 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.015819 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:14.016807 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:14.016810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:14.016813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:14.016826 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:14.016847 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:14.016855 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:14.016912 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:14.017049 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 06:24:14.017128 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 06:24:14.017582 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 678
I20260810 06:24:14.017732 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-0/wal/instance
I20260810 06:24:14.018096 32563 external_mini_cluster.cc:1428] Setting key bc6fe07dbfaacec1871242784fdb9114
I20260810 06:24:14.018100 888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:14.010665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:41045 (local address 127.31.204.193:39979)
0810 06:24:14.011287 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:14.011324 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:14.012549 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:14.013049 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:14.013055 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:14.013134 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:14.013577 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:14.013590 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.014424 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.014427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:14.015398 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:14.015409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.015870 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.015874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:14.016023 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:14.016568 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:24:14.016569 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:14.016605 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:24:14.016664 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:14.016923 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:14.016932 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:14.016938 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 06:24:14.017040 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 06:24:14.017292 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":129,"threads_started":1}
I20260810 06:24:14.019251 884 heartbeater.cc:347] Connected to a master server at 127.31.204.254:41045
I20260810 06:24:14.019467 884 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:14.020041 884 heartbeater.cc:511] Master 127.31.204.254:41045 requested a full tablet report, sending...
I20260810 06:24:14.020391 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.194:0
--local_ip_for_outbound_sockets=127.31.204.194
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:41045
--builtin_ntp_servers=127.31.204.212:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 06:24:14.021538 587 ts_manager.cc:194] Registered new tserver with Master: 59aba2f3323d4024aa1b5144be1a4893 (127.31.204.193:38887)
I20260810 06:24:14.022679 587 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.204.193:39979
W20260810 06:24:14.132704 889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:14.132958 889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:14.133000 889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:14.136976 889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:14.137032 889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:14.137110 889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.194
I20260810 06:24:14.141748 889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.204.194
--webserver_port=0
--tserver_master_addrs=127.31.204.254:41045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.194
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:14.142788 889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:14.144287 889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:14.153460 897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:14.153519 895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:14.153461 903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:14.153661 889 server_base.cc:1034] running on GCE node
I20260810 06:24:14.154206 889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:14.154752 889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:14.155926 889 hybrid_clock.cc:648] HybridClock initialized: now 1786343054155913 us; error 30 us; skew 500 ppm
I20260810 06:24:14.157577 889 webserver.cc:460] Webserver started at http://127.31.204.194:39901/ using document root <none> and password file <none>
I20260810 06:24:14.158149 889 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:14.158222 889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:14.158419 889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:14.160328 889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-1/data/instance:
uuid: "e73e046eea6944969180645f545a731b"
format_stamp: "Formatted at 2026-08-10 06:24:14 on dist-test-slave-q9h9"
server_key: "3940492990a800dfe2bc95772b98862f"
server_key_iv: "8ffe0fc0bb580579f809c584cb913f9b"
server_key_version: "encryptionkey@0"
I20260810 06:24:14.160856 889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance:
uuid: "e73e046eea6944969180645f545a731b"
format_stamp: "Formatted at 2026-08-10 06:24:14 on dist-test-slave-q9h9"
server_key: "3940492990a800dfe2bc95772b98862f"
server_key_iv: "8ffe0fc0bb580579f809c584cb913f9b"
server_key_version: "encryptionkey@0"
I20260810 06:24:14.164335 889 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:24:14.166592 914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:14.167811 889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:24:14.167922 889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-1/wal
uuid: "e73e046eea6944969180645f545a731b"
format_stamp: "Formatted at 2026-08-10 06:24:14 on dist-test-slave-q9h9"
server_key: "3940492990a800dfe2bc95772b98862f"
server_key_iv: "8ffe0fc0bb580579f809c584cb913f9b"
server_key_version: "encryptionkey@0"
I20260810 06:24:14.168005 889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:14.184332 889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:14.184906 889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:14.185058 889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:14.185575 889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:14.186501 889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:14.186547 889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:14.186596 889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:14.186625 889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:14.194790 889 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.194:36903
I20260810 06:24:14.194821 1076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.194:36903 every 8 connection(s)
I20260810 06:24:14.195590 889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-1/data/info.pb
I20260810 06:24:14.197260 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 889
I20260810 06:24:14.197383 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-1/wal/instance
I20260810 06:24:14.197655 32563 external_mini_cluster.cc:1428] Setting key 136a6303ba822af5c896bf5d01b2ac05
I20260810 06:24:14.199492 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.204.195:0
--local_ip_for_outbound_sockets=127.31.204.195
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:41045
--builtin_ntp_servers=127.31.204.212:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 06:24:14.204276 650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:14.197242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.194:35377 (local address 127.31.204.254:41045)
0810 06:24:14.197392 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:14.197397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:14.198483 (+ 1086us) server_negotiation.cc:407] Connection header received
0810 06:24:14.199858 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:14.199862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:14.199916 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:14.199989 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:14.200835 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.201339 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.202480 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.202694 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.202792 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:14.203844 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:14.203847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:14.203849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:14.203860 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:14.203878 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:14.203886 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:14.203940 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:14.204076 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 06:24:14.204137 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 06:24:14.205051 1082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:14.197608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:41045 (local address 127.31.204.194:35377)
0810 06:24:14.198345 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:14.198396 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:14.199660 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:14.200167 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:14.200174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:14.200253 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:14.200706 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:14.200718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.201458 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.201462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:14.202370 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:14.202380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.202832 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.202836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:14.202988 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:14.203575 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:24:14.203577 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:14.203627 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:24:14.203704 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:14.203950 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:14.203960 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:14.203965 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 06:24:14.204068 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 06:24:14.204335 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":132,"threads_started":1}
I20260810 06:24:14.206070 1079 heartbeater.cc:347] Connected to a master server at 127.31.204.254:41045
I20260810 06:24:14.206279 1079 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:14.206728 1079 heartbeater.cc:511] Master 127.31.204.254:41045 requested a full tablet report, sending...
I20260810 06:24:14.207729 587 ts_manager.cc:194] Registered new tserver with Master: e73e046eea6944969180645f545a731b (127.31.204.194:36903)
I20260810 06:24:14.208467 587 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.204.194:35377
W20260810 06:24:14.307526 1085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:14.307763 1085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:14.307799 1085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:14.311673 1085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:14.311722 1085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:14.311772 1085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.204.195
I20260810 06:24:14.316514 1085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.204.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.204.195
--webserver_port=0
--tserver_master_addrs=127.31.204.254:41045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.204.195
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:14.317425 1085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:14.318759 1085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:14.327521 1097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:14.327528 1095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:14.327528 1094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:14.327675 1085 server_base.cc:1034] running on GCE node
I20260810 06:24:14.328253 1085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:14.328814 1085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:14.329972 1085 hybrid_clock.cc:648] HybridClock initialized: now 1786343054329949 us; error 37 us; skew 500 ppm
I20260810 06:24:14.331650 1085 webserver.cc:460] Webserver started at http://127.31.204.195:45875/ using document root <none> and password file <none>
I20260810 06:24:14.332170 1085 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:14.332235 1085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:14.332413 1085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:14.334371 1085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-2/data/instance:
uuid: "3b70736b089d4991b76e3657ac05f8fb"
format_stamp: "Formatted at 2026-08-10 06:24:14 on dist-test-slave-q9h9"
server_key: "8fc493fad2d9d87c6e6c5b9a2ade62f5"
server_key_iv: "0585eb7a5b5be7635552c4ea301538d8"
server_key_version: "encryptionkey@0"
I20260810 06:24:14.334898 1085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance:
uuid: "3b70736b089d4991b76e3657ac05f8fb"
format_stamp: "Formatted at 2026-08-10 06:24:14 on dist-test-slave-q9h9"
server_key: "8fc493fad2d9d87c6e6c5b9a2ade62f5"
server_key_iv: "0585eb7a5b5be7635552c4ea301538d8"
server_key_version: "encryptionkey@0"
I20260810 06:24:14.338385 1085 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:24:14.340566 1104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:14.341677 1085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:14.341789 1085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-2/wal
uuid: "3b70736b089d4991b76e3657ac05f8fb"
format_stamp: "Formatted at 2026-08-10 06:24:14 on dist-test-slave-q9h9"
server_key: "8fc493fad2d9d87c6e6c5b9a2ade62f5"
server_key_iv: "0585eb7a5b5be7635552c4ea301538d8"
server_key_version: "encryptionkey@0"
I20260810 06:24:14.341897 1085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:14.356207 1085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:14.356746 1085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:14.356874 1085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:14.357347 1085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:14.358210 1085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:14.358255 1085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:14.358299 1085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:14.358319 1085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:14.366181 1085 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.195:35061
I20260810 06:24:14.366247 1295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.195:35061 every 8 connection(s)
I20260810 06:24:14.366981 1085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-2/data/info.pb
I20260810 06:24:14.375121 650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:14.368574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:41337 (local address 127.31.204.254:41045)
0810 06:24:14.368742 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:14.368747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:14.369726 (+ 979us) server_negotiation.cc:407] Connection header received
0810 06:24:14.371055 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:14.371058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:14.371106 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:14.371180 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:14.372028 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.372540 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.373570 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.373781 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.373854 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:14.374750 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:14.374753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:14.374755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:14.374768 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:14.374786 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:14.374794 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:14.374859 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:14.374958 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 06:24:14.375008 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 06:24:14.375877 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:14.368974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:41045 (local address 127.31.204.195:41337)
0810 06:24:14.369588 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:14.369624 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:14.370880 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:14.371317 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:14.371323 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:14.371431 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:14.371907 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:14.371920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.372642 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.372645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:14.373478 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:14.373487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.373888 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.373890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:14.374049 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:14.374524 (+ 475us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:24:14.374526 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:14.374562 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:24:14.374620 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:14.374865 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:14.374869 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:14.374872 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 06:24:14.374954 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 06:24:14.375150 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":124,"threads_started":1}
I20260810 06:24:14.375967 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 1085
I20260810 06:24:14.376075 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-2/wal/instance
I20260810 06:24:14.376396 32563 external_mini_cluster.cc:1428] Setting key a5eeb9d0f8f3f256444671b000f448df
I20260810 06:24:14.376969 1296 heartbeater.cc:347] Connected to a master server at 127.31.204.254:41045
I20260810 06:24:14.377190 1296 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:14.377705 1296 heartbeater.cc:511] Master 127.31.204.254:41045 requested a full tablet report, sending...
I20260810 06:24:14.378728 587 ts_manager.cc:194] Registered new tserver with Master: 3b70736b089d4991b76e3657ac05f8fb (127.31.204.195:35061)
I20260810 06:24:14.379220 587 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.204.195:41337
I20260810 06:24:14.389776 32563 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:24:14.394680 650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:14.391658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45788 (local address 127.31.204.254:41045)
0810 06:24:14.391754 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:14.391757 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:14.391957 (+ 200us) server_negotiation.cc:407] Connection header received
0810 06:24:14.392043 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:14.392045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:14.392079 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:14.392138 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:14.392352 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.392813 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.393535 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.393720 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.393788 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:14.394406 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:14.394409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:14.394411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:14.394421 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:14.394438 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:14.394445 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:14.394495 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:14.394550 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 06:24:14.394591 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 06:24:14.396939 587 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:24:14.399725 587 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:24:14.412907 1320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:14.407061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.195:35061 (local address 127.0.0.1:47092)
0810 06:24:14.407812 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:14.407824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:14.407892 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:14.409265 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:14.409268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:14.409286 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:14.409371 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:14.409377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.410445 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.410448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:14.411467 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:14.411476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.412413 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.412417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:14.412517 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:14.412552 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 06:24:14.412657 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 06:24:14.412763 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260810 06:24:14.412936 1321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:14.407665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47092 (local address 127.31.204.195:35061)
0810 06:24:14.408884 (+ 1219us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:14.408895 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:14.408916 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:24:14.408981 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:14.408985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:14.409153 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:14.409276 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:14.409474 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.410359 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.411579 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.412289 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.412413 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:14.412585 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:14.412685 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 06:24:14.412778 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1120,"thread_start_us":92,"threads_started":1}
I20260810 06:24:14.413226 1319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:14.407122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.31.204.194:36903)
0810 06:24:14.408372 (+ 1250us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:14.408383 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:14.408408 (+ 25us) server_negotiation.cc:407] Connection header received
0810 06:24:14.408481 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:14.408487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:14.408670 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:14.408878 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:14.409077 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.410179 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.411820 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.412320 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.412425 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:14.412593 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:14.412980 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 06:24:14.413088 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1129,"thread_start_us":159,"threads_started":1}
I20260810 06:24:14.413235 1318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:14.407329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:38887 (local address 127.0.0.1:40790)
0810 06:24:14.408036 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:14.408047 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:14.408120 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:14.408577 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:14.408583 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:14.408602 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:14.408736 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:14.408743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.410169 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.410172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:14.411674 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:14.411684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.412872 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.412876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:14.412962 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:14.412986 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 06:24:14.413054 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 06:24:14.413121 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":130,"thread_start_us":82,"threads_started":1}
I20260810 06:24:14.413484 1311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:14.406954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:36903 (local address 127.0.0.1:47722)
0810 06:24:14.407644 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:14.407698 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:14.407841 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:14.408848 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:14.408852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:14.408871 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:14.408967 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:14.408973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.410277 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.410280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:14.411673 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:14.411684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.412653 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.412657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:14.412733 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:14.412753 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 06:24:14.412829 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 06:24:14.412881 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":57,"thread_start_us":53,"threads_started":1}
I20260810 06:24:14.413661 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:14.407730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40790 (local address 127.31.204.193:38887)
0810 06:24:14.408132 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:14.408143 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:14.408168 (+ 25us) server_negotiation.cc:407] Connection header received
0810 06:24:14.408245 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:14.408250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:14.408431 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:14.408588 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:14.408878 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.410060 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.411794 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.412545 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.412660 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:14.412806 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:14.413385 (+ 579us) server_negotiation.cc:299] Negotiation successful
0810 06:24:14.413483 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":112,"threads_started":1}
I20260810 06:24:14.416253 780 tablet_service.cc:1467] Processing CreateTablet for tablet 86dda4a289304095b72b6e60910a85f5 (DEFAULT_TABLE table=test-table [id=bcb65819d82543ebb1f290e6cbd36250]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:24:14.416311 1178 tablet_service.cc:1467] Processing CreateTablet for tablet 86dda4a289304095b72b6e60910a85f5 (DEFAULT_TABLE table=test-table [id=bcb65819d82543ebb1f290e6cbd36250]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:24:14.416311 979 tablet_service.cc:1467] Processing CreateTablet for tablet 86dda4a289304095b72b6e60910a85f5 (DEFAULT_TABLE table=test-table [id=bcb65819d82543ebb1f290e6cbd36250]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:24:14.417106 780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86dda4a289304095b72b6e60910a85f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:14.417259 979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86dda4a289304095b72b6e60910a85f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:14.417269 1178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86dda4a289304095b72b6e60910a85f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:14.423576 1327 tablet_bootstrap.cc:492] T 86dda4a289304095b72b6e60910a85f5 P 59aba2f3323d4024aa1b5144be1a4893: Bootstrap starting.
I20260810 06:24:14.424284 1328 tablet_bootstrap.cc:492] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b: Bootstrap starting.
I20260810 06:24:14.425887 1327 tablet_bootstrap.cc:654] T 86dda4a289304095b72b6e60910a85f5 P 59aba2f3323d4024aa1b5144be1a4893: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:14.426222 1331 tablet_bootstrap.cc:492] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb: Bootstrap starting.
I20260810 06:24:14.426394 1328 tablet_bootstrap.cc:654] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:14.426803 1327 log.cc:824] T 86dda4a289304095b72b6e60910a85f5 P 59aba2f3323d4024aa1b5144be1a4893: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:14.427191 1328 log.cc:824] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:14.428030 1331 tablet_bootstrap.cc:654] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:14.428809 1331 log.cc:824] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:14.428858 1327 tablet_bootstrap.cc:492] T 86dda4a289304095b72b6e60910a85f5 P 59aba2f3323d4024aa1b5144be1a4893: No bootstrap required, opened a new log
I20260810 06:24:14.429044 1327 ts_tablet_manager.cc:1397] T 86dda4a289304095b72b6e60910a85f5 P 59aba2f3323d4024aa1b5144be1a4893: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:24:14.429104 1328 tablet_bootstrap.cc:492] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b: No bootstrap required, opened a new log
I20260810 06:24:14.429281 1328 ts_tablet_manager.cc:1397] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 06:24:14.430874 1331 tablet_bootstrap.cc:492] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb: No bootstrap required, opened a new log
I20260810 06:24:14.431051 1331 ts_tablet_manager.cc:1397] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:24:14.432286 1327 raft_consensus.cc:348] T 86dda4a289304095b72b6e60910a85f5 P 59aba2f3323d4024aa1b5144be1a4893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e046eea6944969180645f545a731b" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36903 } } peers { permanent_uuid: "3b70736b089d4991b76e3657ac05f8fb" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 35061 } } peers { permanent_uuid: "59aba2f3323d4024aa1b5144be1a4893" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38887 } }
I20260810 06:24:14.432492 1327 raft_consensus.cc:374] T 86dda4a289304095b72b6e60910a85f5 P 59aba2f3323d4024aa1b5144be1a4893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:14.432554 1327 raft_consensus.cc:729] T 86dda4a289304095b72b6e60910a85f5 P 59aba2f3323d4024aa1b5144be1a4893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59aba2f3323d4024aa1b5144be1a4893, State: Initialized, Role: FOLLOWER
I20260810 06:24:14.432919 1327 consensus_queue.cc:260] T 86dda4a289304095b72b6e60910a85f5 P 59aba2f3323d4024aa1b5144be1a4893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e046eea6944969180645f545a731b" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36903 } } peers { permanent_uuid: "3b70736b089d4991b76e3657ac05f8fb" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 35061 } } peers { permanent_uuid: "59aba2f3323d4024aa1b5144be1a4893" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38887 } }
I20260810 06:24:14.433291 1328 raft_consensus.cc:348] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e046eea6944969180645f545a731b" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36903 } } peers { permanent_uuid: "3b70736b089d4991b76e3657ac05f8fb" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 35061 } } peers { permanent_uuid: "59aba2f3323d4024aa1b5144be1a4893" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38887 } }
I20260810 06:24:14.433529 1328 raft_consensus.cc:374] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:14.433561 884 heartbeater.cc:503] Master 127.31.204.254:41045 was elected leader, sending a full tablet report...
I20260810 06:24:14.433595 1328 raft_consensus.cc:729] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e73e046eea6944969180645f545a731b, State: Initialized, Role: FOLLOWER
I20260810 06:24:14.433843 1327 ts_tablet_manager.cc:1428] T 86dda4a289304095b72b6e60910a85f5 P 59aba2f3323d4024aa1b5144be1a4893: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 06:24:14.434005 1328 consensus_queue.cc:260] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e046eea6944969180645f545a731b" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36903 } } peers { permanent_uuid: "3b70736b089d4991b76e3657ac05f8fb" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 35061 } } peers { permanent_uuid: "59aba2f3323d4024aa1b5144be1a4893" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38887 } }
I20260810 06:24:14.434615 1079 heartbeater.cc:503] Master 127.31.204.254:41045 was elected leader, sending a full tablet report...
I20260810 06:24:14.434995 1328 ts_tablet_manager.cc:1428] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260810 06:24:14.436152 1331 raft_consensus.cc:348] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e046eea6944969180645f545a731b" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36903 } } peers { permanent_uuid: "3b70736b089d4991b76e3657ac05f8fb" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 35061 } } peers { permanent_uuid: "59aba2f3323d4024aa1b5144be1a4893" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38887 } }
I20260810 06:24:14.436405 1331 raft_consensus.cc:374] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:14.436463 1331 raft_consensus.cc:729] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b70736b089d4991b76e3657ac05f8fb, State: Initialized, Role: FOLLOWER
I20260810 06:24:14.437394 1331 consensus_queue.cc:260] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e046eea6944969180645f545a731b" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36903 } } peers { permanent_uuid: "3b70736b089d4991b76e3657ac05f8fb" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 35061 } } peers { permanent_uuid: "59aba2f3323d4024aa1b5144be1a4893" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38887 } }
I20260810 06:24:14.438000 1296 heartbeater.cc:503] Master 127.31.204.254:41045 was elected leader, sending a full tablet report...
I20260810 06:24:14.438298 1331 ts_tablet_manager.cc:1428] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260810 06:24:14.446358 1080 tablet.cc:2369] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:24:14.448158 1340 raft_consensus.cc:482] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:24:14.448279 1340 raft_consensus.cc:504] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e046eea6944969180645f545a731b" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36903 } } peers { permanent_uuid: "3b70736b089d4991b76e3657ac05f8fb" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 35061 } } peers { permanent_uuid: "59aba2f3323d4024aa1b5144be1a4893" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38887 } }
I20260810 06:24:14.449136 1340 leader_election.cc:288] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e73e046eea6944969180645f545a731b (127.31.204.194:36903), 59aba2f3323d4024aa1b5144be1a4893 (127.31.204.193:38887)
I20260810 06:24:14.452376 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:14.449285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.194:36903 (local address 127.31.204.195:45689)
0810 06:24:14.449483 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:14.449499 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:14.449575 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:14.449791 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:14.449793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:14.449809 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:14.449911 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:14.449916 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.450627 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.450631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:14.451396 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:14.451405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.452001 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.452004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:14.452098 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:14.452116 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 06:24:14.452177 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 06:24:14.452240 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":2}
I20260810 06:24:14.452374 1319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:14.449408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:45689 (local address 127.31.204.194:36903)
0810 06:24:14.449518 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:14.449523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:14.449537 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:24:14.449669 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:14.449673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:14.449715 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:14.449782 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:14.450001 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.450523 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.451508 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.451895 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.451997 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:14.452045 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:14.452200 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 06:24:14.452268 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:24:14.452589 1342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:14.449367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:38887 (local address 127.31.204.195:43995)
0810 06:24:14.449702 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:14.449714 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:14.449808 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:14.450044 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:14.450047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:14.450062 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:24:14.450186 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:14.450192 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.450862 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.450866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:14.451614 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:14.451622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.452206 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.452210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:14.452304 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:14.452325 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 06:24:14.452388 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 06:24:14.452451 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":105,"threads_started":1}
I20260810 06:24:14.452579 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:14.449460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.204.195:43995 (local address 127.31.204.193:38887)
0810 06:24:14.449579 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:14.449583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:14.449791 (+ 208us) server_negotiation.cc:407] Connection header received
0810 06:24:14.449907 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:14.449909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:14.449947 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:14.450004 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:24:14.450287 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.450764 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.451733 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.452104 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.452206 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:14.452252 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:14.452415 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 06:24:14.452482 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 06:24:14.452970 1017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86dda4a289304095b72b6e60910a85f5" candidate_uuid: "3b70736b089d4991b76e3657ac05f8fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e73e046eea6944969180645f545a731b" is_pre_election: true
I20260810 06:24:14.453184 810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86dda4a289304095b72b6e60910a85f5" candidate_uuid: "3b70736b089d4991b76e3657ac05f8fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59aba2f3323d4024aa1b5144be1a4893" is_pre_election: true
I20260810 06:24:14.453238 1017 raft_consensus.cc:2454] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b70736b089d4991b76e3657ac05f8fb in term 0.
I20260810 06:24:14.453426 810 raft_consensus.cc:2454] T 86dda4a289304095b72b6e60910a85f5 P 59aba2f3323d4024aa1b5144be1a4893 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b70736b089d4991b76e3657ac05f8fb in term 0.
I20260810 06:24:14.453788 1110 leader_election.cc:302] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b70736b089d4991b76e3657ac05f8fb, 59aba2f3323d4024aa1b5144be1a4893, e73e046eea6944969180645f545a731b; no voters:
I20260810 06:24:14.454041 1340 raft_consensus.cc:2781] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:24:14.454131 1340 raft_consensus.cc:482] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:24:14.454183 1340 raft_consensus.cc:3037] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:14.455367 1340 raft_consensus.cc:504] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e046eea6944969180645f545a731b" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36903 } } peers { permanent_uuid: "3b70736b089d4991b76e3657ac05f8fb" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 35061 } } peers { permanent_uuid: "59aba2f3323d4024aa1b5144be1a4893" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38887 } }
I20260810 06:24:14.455678 1340 leader_election.cc:288] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [CANDIDATE]: Term 1 election: Requested vote from peers e73e046eea6944969180645f545a731b (127.31.204.194:36903), 59aba2f3323d4024aa1b5144be1a4893 (127.31.204.193:38887)
I20260810 06:24:14.455963 1017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86dda4a289304095b72b6e60910a85f5" candidate_uuid: "3b70736b089d4991b76e3657ac05f8fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e73e046eea6944969180645f545a731b"
I20260810 06:24:14.455977 810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86dda4a289304095b72b6e60910a85f5" candidate_uuid: "3b70736b089d4991b76e3657ac05f8fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59aba2f3323d4024aa1b5144be1a4893"
I20260810 06:24:14.456068 810 raft_consensus.cc:3037] T 86dda4a289304095b72b6e60910a85f5 P 59aba2f3323d4024aa1b5144be1a4893 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:14.456065 1017 raft_consensus.cc:3037] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:14.456982 1017 raft_consensus.cc:2454] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b70736b089d4991b76e3657ac05f8fb in term 1.
I20260810 06:24:14.457010 810 raft_consensus.cc:2454] T 86dda4a289304095b72b6e60910a85f5 P 59aba2f3323d4024aa1b5144be1a4893 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b70736b089d4991b76e3657ac05f8fb in term 1.
I20260810 06:24:14.457255 1110 leader_election.cc:302] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b70736b089d4991b76e3657ac05f8fb, e73e046eea6944969180645f545a731b; no voters:
I20260810 06:24:14.457417 1340 raft_consensus.cc:2781] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:14.457612 1340 raft_consensus.cc:686] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [term 1 LEADER]: Becoming Leader. State: Replica: 3b70736b089d4991b76e3657ac05f8fb, State: Running, Role: LEADER
I20260810 06:24:14.457887 1340 consensus_queue.cc:237] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e046eea6944969180645f545a731b" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36903 } } peers { permanent_uuid: "3b70736b089d4991b76e3657ac05f8fb" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 35061 } } peers { permanent_uuid: "59aba2f3323d4024aa1b5144be1a4893" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38887 } }
I20260810 06:24:14.460682 586 catalog_manager.cc:5495] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b70736b089d4991b76e3657ac05f8fb (127.31.204.195). New cstate: current_term: 1 leader_uuid: "3b70736b089d4991b76e3657ac05f8fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e046eea6944969180645f545a731b" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b70736b089d4991b76e3657ac05f8fb" member_type: VOTER last_known_addr { host: "127.31.204.195" port: 35061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59aba2f3323d4024aa1b5144be1a4893" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38887 } health_report { overall_health: UNKNOWN } } }
I20260810 06:24:14.474381 1321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:14.471485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47100 (local address 127.31.204.195:35061)
0810 06:24:14.471595 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:14.471598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:14.471610 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:24:14.471729 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:14.471731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:14.471783 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:14.471857 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:24:14.472058 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.472497 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.473268 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:14.473456 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:14.473519 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:14.473726 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:24:14.474085 (+ 359us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:24:14.474163 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:14.474209 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 06:24:14.474247 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:24:14.479202 810 raft_consensus.cc:1261] T 86dda4a289304095b72b6e60910a85f5 P 59aba2f3323d4024aa1b5144be1a4893 [term 1 FOLLOWER]: Refusing update from remote peer 3b70736b089d4991b76e3657ac05f8fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:24:14.479200 1017 raft_consensus.cc:1261] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b [term 1 FOLLOWER]: Refusing update from remote peer 3b70736b089d4991b76e3657ac05f8fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:24:14.479794 1343 consensus_queue.cc:1035] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e73e046eea6944969180645f545a731b" member_type: VOTER last_known_addr { host: "127.31.204.194" port: 36903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:24:14.479934 1340 consensus_queue.cc:1035] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "59aba2f3323d4024aa1b5144be1a4893" member_type: VOTER last_known_addr { host: "127.31.204.193" port: 38887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:24:14.488343 1349 mvcc.cc:205] Tried to move back new op lower bound from 7316861151141154816 to 7316861151061917696. Current Snapshot: MvccSnapshot[applied={T|T < 7316861151141154816}]
I20260810 06:24:14.495167 588 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:24:14.495316 588 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:24:14.497824 588 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d3dc3a5255b4ff5ba80ab357d286c7a: Sending DeleteTablet for 3 replicas of tablet 86dda4a289304095b72b6e60910a85f5
I20260810 06:24:14.498488 979 tablet_service.cc:1514] Processing DeleteTablet for tablet 86dda4a289304095b72b6e60910a85f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:24:14 UTC) from {username='slave'} at 127.0.0.1:47722
I20260810 06:24:14.498654 1178 tablet_service.cc:1514] Processing DeleteTablet for tablet 86dda4a289304095b72b6e60910a85f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:24:14 UTC) from {username='slave'} at 127.0.0.1:47092
I20260810 06:24:14.498904 780 tablet_service.cc:1514] Processing DeleteTablet for tablet 86dda4a289304095b72b6e60910a85f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:24:14 UTC) from {username='slave'} at 127.0.0.1:40790
I20260810 06:24:14.498950 1358 tablet_replica.cc:330] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b: stopping tablet replica
I20260810 06:24:14.499207 1358 raft_consensus.cc:2229] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:24:14.499414 1360 tablet_replica.cc:330] T 86dda4a289304095b72b6e60910a85f5 P 59aba2f3323d4024aa1b5144be1a4893: stopping tablet replica
I20260810 06:24:14.499455 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 678
I20260810 06:24:14.499493 1358 raft_consensus.cc:2258] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:24:14.512360 1358 ts_tablet_manager.cc:1905] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:24:14.514597 1358 ts_tablet_manager.cc:1918] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:24:14.514685 1358 log.cc:1196] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-1/wal/wals/86dda4a289304095b72b6e60910a85f5
I20260810 06:24:14.514973 1358 ts_tablet_manager.cc:1939] T 86dda4a289304095b72b6e60910a85f5 P e73e046eea6944969180645f545a731b: Deleting consensus metadata
I20260810 06:24:14.515981 569 catalog_manager.cc:4841] TS e73e046eea6944969180645f545a731b (127.31.204.194:36903): tablet 86dda4a289304095b72b6e60910a85f5 (table test-table [id=bcb65819d82543ebb1f290e6cbd36250]) successfully deleted
I20260810 06:24:14.518206 1359 tablet_replica.cc:330] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb: stopping tablet replica
I20260810 06:24:14.518460 1359 raft_consensus.cc:2229] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:24:14.518774 1359 raft_consensus.cc:2258] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:24:14.519829 1359 ts_tablet_manager.cc:1905] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:24:14.521775 1359 ts_tablet_manager.cc:1918] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:24:14.521867 1359 log.cc:1196] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb: Deleting WAL directory at /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786342803670820-32563-0/minicluster-data/ts-2/wal/wals/86dda4a289304095b72b6e60910a85f5
W20260810 06:24:14.521919 567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.204.193:38887 (error 108)
I20260810 06:24:14.522166 1359 ts_tablet_manager.cc:1939] T 86dda4a289304095b72b6e60910a85f5 P 3b70736b089d4991b76e3657ac05f8fb: Deleting consensus metadata
I20260810 06:24:14.522970 1311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:14.522682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.193:38887 (local address 127.0.0.1:40798)
0810 06:24:14.522818 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:14.522890 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.193:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 06:24:14.523003 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 889
I20260810 06:24:14.523149 565 catalog_manager.cc:4841] TS 3b70736b089d4991b76e3657ac05f8fb (127.31.204.195:35061): tablet 86dda4a289304095b72b6e60910a85f5 (table test-table [id=bcb65819d82543ebb1f290e6cbd36250]) successfully deleted
W20260810 06:24:14.532001 567 catalog_manager.cc:4584] TS 59aba2f3323d4024aa1b5144be1a4893 (127.31.204.193:38887): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 86dda4a289304095b72b6e60910a85f5: Network error: Client connection negotiation failed: client connection to 127.31.204.193:38887: connect: Connection refused (error 111)
I20260810 06:24:14.548278 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 1085
I20260810 06:24:14.557930 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 544
2026-08-10T06:24:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (971 ms)
[----------] 6 tests from EncryptionPolicyTest (10664 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:14 dist-test-slave-q9h9 krb5kdc[1368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:14 dist-test-slave-q9h9 krb5kdc[1368](info): set up 2 sockets
Aug 10 06:24:14 dist-test-slave-q9h9 krb5kdc[1368](info): commencing operation
krb5kdc: starting...
W20260810 06:24:15.699304 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:15 dist-test-slave-q9h9 krb5kdc[1368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:24:15.813999 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:15.920550 1385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:15.920790 1385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:15.920835 1385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:15.924707 1385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:15.924762 1385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:15.924783 1385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:15.924813 1385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:15.924839 1385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:15.929864 1385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46593
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:15.930886 1385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:15.931941 1385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:15.939949 1391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:15.940039 1392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:15.940099 1385 server_base.cc:1034] running on GCE node
W20260810 06:24:15.939939 1394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:15.940666 1385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:15.941658 1385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:15.942809 1385 hybrid_clock.cc:648] HybridClock initialized: now 1786343055942796 us; error 24 us; skew 500 ppm
Aug 10 06:24:15 dist-test-slave-q9h9 krb5kdc[1368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:15.945307 1385 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:24:15.946144 1385 webserver.cc:460] Webserver started at http://127.0.0.1:42375/ using document root <none> and password file <none>
I20260810 06:24:15.946645 1385 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:15.946708 1385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:15.946904 1385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:15.948809 1385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "3a84fbaf7a494a4a910d73bac8d34555"
format_stamp: "Formatted at 2026-08-10 06:24:15 on dist-test-slave-q9h9"
server_key: "5599bb7320ed9c8da55a7f302942a950"
server_key_iv: "4178601d3319e1b1044b59fffa834c2c"
server_key_version: "encryptionkey@0"
I20260810 06:24:15.949339 1385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "3a84fbaf7a494a4a910d73bac8d34555"
format_stamp: "Formatted at 2026-08-10 06:24:15 on dist-test-slave-q9h9"
server_key: "5599bb7320ed9c8da55a7f302942a950"
server_key_iv: "4178601d3319e1b1044b59fffa834c2c"
server_key_version: "encryptionkey@0"
I20260810 06:24:15.952760 1385 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260810 06:24:15.954955 1402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:15.956091 1385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:24:15.956202 1385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "3a84fbaf7a494a4a910d73bac8d34555"
format_stamp: "Formatted at 2026-08-10 06:24:15 on dist-test-slave-q9h9"
server_key: "5599bb7320ed9c8da55a7f302942a950"
server_key_iv: "4178601d3319e1b1044b59fffa834c2c"
server_key_version: "encryptionkey@0"
I20260810 06:24:15.956296 1385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:15.972512 1385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:15.975039 1385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:15.975200 1385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:15.981737 1385 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46593
I20260810 06:24:15.981763 1489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46593 every 8 connection(s)
I20260810 06:24:15.982657 1385 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:15.985167 1493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:15.989403 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 1385
I20260810 06:24:15.989544 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:15.989853 32563 external_mini_cluster.cc:1428] Setting key 7fb391590ac7b6a78f70551a0368837a
I20260810 06:24:15.990746 1493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555: Bootstrap starting.
I20260810 06:24:15.992982 1493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:15.993680 1493 log.cc:824] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:15.995503 1493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555: No bootstrap required, opened a new log
Aug 10 06:24:15 dist-test-slave-q9h9 krb5kdc[1368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 06:24:15.999418 1493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a84fbaf7a494a4a910d73bac8d34555" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260810 06:24:15.999616 1493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:15.999651 1493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a84fbaf7a494a4a910d73bac8d34555, State: Initialized, Role: FOLLOWER
I20260810 06:24:16.000073 1493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a84fbaf7a494a4a910d73bac8d34555" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260810 06:24:16.000238 1493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:16.000305 1493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:16.000401 1493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:16.001363 1493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a84fbaf7a494a4a910d73bac8d34555" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260810 06:24:16.001730 1493 leader_election.cc:302] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a84fbaf7a494a4a910d73bac8d34555; no voters:
I20260810 06:24:16.002056 1493 leader_election.cc:288] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:16.002140 1499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:16.002353 1499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555 [term 1 LEADER]: Becoming Leader. State: Replica: 3a84fbaf7a494a4a910d73bac8d34555, State: Running, Role: LEADER
I20260810 06:24:16.002693 1499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a84fbaf7a494a4a910d73bac8d34555" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260810 06:24:16.003230 1493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:16.004452 1497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:15.991168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52609 (local address 127.0.0.1:46593)
0810 06:24:15.991584 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:15.991598 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:15.991632 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:24:15.992226 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:15.992241 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:15.992545 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:15.992837 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:15.993627 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:15.994812 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:15.995651 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:15.996314 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:15.996481 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:15.999029 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:15.999034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:15.999046 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:15.999071 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:16.000960 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:16.001437 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:16.001443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:16.001449 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:16.001583 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:16.001911 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:16.001915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:16.001916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:16.002436 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:16.002653 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:16.003358 (+ 705us) server_negotiation.cc:299] Negotiation successful
0810 06:24:16.003570 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":126,"threads_started":1}
I20260810 06:24:16.005203 1500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a84fbaf7a494a4a910d73bac8d34555" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a84fbaf7a494a4a910d73bac8d34555" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } } }
I20260810 06:24:16.005434 1501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a84fbaf7a494a4a910d73bac8d34555. Latest consensus state: current_term: 1 leader_uuid: "3a84fbaf7a494a4a910d73bac8d34555" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a84fbaf7a494a4a910d73bac8d34555" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } } }
I20260810 06:24:16.005489 1500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:16.005546 1501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:16.007637 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 06:24:16.008252 1518 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:24:16.008328 1518 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:24:16.008421 1519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:16.009308 1519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:16.014714 1519 catalog_manager.cc:1347] Generated new cluster ID: 0c6eb88a865a4a6290896e0264144713
I20260810 06:24:16.014776 1519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:16.019483 1497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:16.010441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60297 (local address 127.0.0.1:46593)
0810 06:24:16.010535 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:16.010539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:16.011342 (+ 803us) server_negotiation.cc:407] Connection header received
0810 06:24:16.011451 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:16.011454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:16.011495 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:16.011572 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:16.012354 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:16.013304 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:16.014104 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:16.014339 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:16.014424 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:16.016097 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:16.016101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:16.016103 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:16.016129 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:16.017729 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:16.018260 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:16.018264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:16.018266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:16.018350 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:16.018637 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:16.018640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:16.018642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:16.018839 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:16.018974 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:16.019203 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 06:24:16.019296 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 06:24:16.022197 1519 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:16.023021 1519 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:16.026866 1519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a84fbaf7a494a4a910d73bac8d34555: Generated new TSK 0
I20260810 06:24:16.027526 1519 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:16.036187 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 1385
W20260810 06:24:16.054436 1366 connection.cc:535] client connection to 127.0.0.1:46593 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46593): Connection reset by peer (error 104)
2026-08-10T06:24:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1486 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:16 dist-test-slave-q9h9 krb5kdc[1533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:16 dist-test-slave-q9h9 krb5kdc[1533](info): set up 2 sockets
Aug 10 06:24:16 dist-test-slave-q9h9 krb5kdc[1533](info): commencing operation
krb5kdc: starting...
W20260810 06:24:17.189868 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:17 dist-test-slave-q9h9 krb5kdc[1533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:24:17.298623 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42965
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:17.404937 1551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:17.405181 1551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:17.405243 1551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:17.409155 1551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:17.409214 1551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:17.409240 1551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:17.409263 1551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:17.409292 1551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:17.414394 1551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42965
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:17.415395 1551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:17.416388 1551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:17.424576 1560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:17.424635 1551 server_base.cc:1034] running on GCE node
W20260810 06:24:17.424549 1562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:17.424585 1564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:17.425318 1551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:17.426417 1551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:17.427558 1551 hybrid_clock.cc:648] HybridClock initialized: now 1786343057427533 us; error 40 us; skew 500 ppm
Aug 10 06:24:17 dist-test-slave-q9h9 krb5kdc[1533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:17.430235 1551 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:24:17.431059 1551 webserver.cc:460] Webserver started at http://127.0.0.1:42519/ using document root <none> and password file <none>
I20260810 06:24:17.431562 1551 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:17.431627 1551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:17.431838 1551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:17.433887 1551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "c25e3a1f83684650a47a86af520dceab"
format_stamp: "Formatted at 2026-08-10 06:24:17 on dist-test-slave-q9h9"
server_key: "f297a3bba0182a2aa4770a23c9b277c8"
server_key_iv: "965b7467006736089d7d21bcca7a2290"
server_key_version: "encryptionkey@0"
I20260810 06:24:17.434458 1551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "c25e3a1f83684650a47a86af520dceab"
format_stamp: "Formatted at 2026-08-10 06:24:17 on dist-test-slave-q9h9"
server_key: "f297a3bba0182a2aa4770a23c9b277c8"
server_key_iv: "965b7467006736089d7d21bcca7a2290"
server_key_version: "encryptionkey@0"
I20260810 06:24:17.437994 1551 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:24:17.440267 1575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:17.441402 1551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:17.441512 1551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "c25e3a1f83684650a47a86af520dceab"
format_stamp: "Formatted at 2026-08-10 06:24:17 on dist-test-slave-q9h9"
server_key: "f297a3bba0182a2aa4770a23c9b277c8"
server_key_iv: "965b7467006736089d7d21bcca7a2290"
server_key_version: "encryptionkey@0"
I20260810 06:24:17.441598 1551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:17.454599 1551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:17.457459 1551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:17.457603 1551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:17.463898 1551 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42965
I20260810 06:24:17.463961 1657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42965 every 8 connection(s)
I20260810 06:24:17.464783 1551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:17.467446 1661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:17.472783 1661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab: Bootstrap starting.
I20260810 06:24:17.473691 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 1551
I20260810 06:24:17.473834 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:17.474211 32563 external_mini_cluster.cc:1428] Setting key d8bd89918a3200008e5d2009e3985de2
I20260810 06:24:17.475665 1661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:17.476377 1661 log.cc:824] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:17.478394 1661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab: No bootstrap required, opened a new log
I20260810 06:24:17.482431 1661 raft_consensus.cc:348] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c25e3a1f83684650a47a86af520dceab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42965 } }
I20260810 06:24:17.482659 1661 raft_consensus.cc:374] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:17.482723 1661 raft_consensus.cc:729] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25e3a1f83684650a47a86af520dceab, State: Initialized, Role: FOLLOWER
I20260810 06:24:17.482745 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:17.475285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37389 (local address 127.0.0.1:42965)
0810 06:24:17.475697 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:17.475712 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:17.475746 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:24:17.476397 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:17.476414 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:17.476817 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:17.477180 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:17.477506 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:17.478522 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:17.479673 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:17.480119 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:17.480232 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:17.481104 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:17.481109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:17.481120 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:17.481483 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:17.481536 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:17.481557 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:17.481652 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:17.481844 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 06:24:17.481985 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":110,"threads_started":1}
I20260810 06:24:17.483198 1661 consensus_queue.cc:260] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c25e3a1f83684650a47a86af520dceab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42965 } }
I20260810 06:24:17.483356 1661 raft_consensus.cc:388] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:17.483419 1661 raft_consensus.cc:482] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:17.483510 1661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:17.484375 1661 raft_consensus.cc:504] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c25e3a1f83684650a47a86af520dceab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42965 } }
I20260810 06:24:17.484730 1661 leader_election.cc:302] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c25e3a1f83684650a47a86af520dceab; no voters:
I20260810 06:24:17.485024 1661 leader_election.cc:288] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:17.485109 1668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:17.485339 1668 raft_consensus.cc:686] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab [term 1 LEADER]: Becoming Leader. State: Replica: c25e3a1f83684650a47a86af520dceab, State: Running, Role: LEADER
I20260810 06:24:17.485612 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:24:17.485697 1668 consensus_queue.cc:237] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c25e3a1f83684650a47a86af520dceab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42965 } }
I20260810 06:24:17.486323 1661 sys_catalog.cc:564] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:17.487191 1669 sys_catalog.cc:455] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c25e3a1f83684650a47a86af520dceab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c25e3a1f83684650a47a86af520dceab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42965 } } }
I20260810 06:24:17.487310 1669 sys_catalog.cc:458] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:17.487628 1670 sys_catalog.cc:455] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab [sys.catalog]: SysCatalogTable state changed. Reason: New leader c25e3a1f83684650a47a86af520dceab. Latest consensus state: current_term: 1 leader_uuid: "c25e3a1f83684650a47a86af520dceab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c25e3a1f83684650a47a86af520dceab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42965 } } }
I20260810 06:24:17.487705 1670 sys_catalog.cc:458] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:17.487811 1686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:17.491081 1686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:17.493188 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:17.488839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60851 (local address 127.0.0.1:42965)
0810 06:24:17.488961 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:17.488965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:17.489171 (+ 206us) server_negotiation.cc:407] Connection header received
0810 06:24:17.489305 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:17.489308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:17.489348 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:17.489406 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:17.489673 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:17.490247 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:17.491457 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:17.491738 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:17.491797 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:17.492773 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:17.492777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:17.492780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:17.492792 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:17.492815 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:17.492825 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:17.492894 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:17.493020 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 06:24:17.493060 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 06:24:17.497030 1686 catalog_manager.cc:1347] Generated new cluster ID: 8786d7f756124ea98a587855c89d0307
I20260810 06:24:17.497082 1686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:17.518713 1686 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:17.519632 1686 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:17.524552 1686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c25e3a1f83684650a47a86af520dceab: Generated new TSK 0
I20260810 06:24:17.525184 1686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:17.537539 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 1551
W20260810 06:24:17.547740 1531 connection.cc:535] client connection to 127.0.0.1:42965 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42965): Connection reset by peer (error 104)
2026-08-10T06:24:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1493 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:17 dist-test-slave-q9h9 krb5kdc[1704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:17 dist-test-slave-q9h9 krb5kdc[1704](info): set up 2 sockets
Aug 10 06:24:17 dist-test-slave-q9h9 krb5kdc[1704](info): commencing operation
krb5kdc: starting...
W20260810 06:24:18.692548 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:18 dist-test-slave-q9h9 krb5kdc[1704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:24:18.801810 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:18.906457 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:18.906682 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:18.906735 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:18.910597 1723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:18.910647 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:18.910674 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:18.910696 1723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:18.910724 1723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:18.915632 1723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46169
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:18.916520 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:18.917410 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:18.925158 1740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:18.925148 1733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:18.925158 1732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:18.925263 1723 server_base.cc:1034] running on GCE node
I20260810 06:24:18.925891 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:18.926872 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:18.928027 1723 hybrid_clock.cc:648] HybridClock initialized: now 1786343058928010 us; error 32 us; skew 500 ppm
Aug 10 06:24:18 dist-test-slave-q9h9 krb5kdc[1704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:18.930604 1723 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:24:18.931352 1723 webserver.cc:460] Webserver started at http://127.0.0.1:45277/ using document root <none> and password file <none>
I20260810 06:24:18.931834 1723 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:18.931896 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:18.932065 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:18.933878 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "0294d374c1b04df7b6e25c703c3c9e38"
format_stamp: "Formatted at 2026-08-10 06:24:18 on dist-test-slave-q9h9"
server_key: "768c45853f390b3198e5af38151f34bd"
server_key_iv: "5d5ccb99cb8ba0b189c843a020e1ae8b"
server_key_version: "encryptionkey@0"
I20260810 06:24:18.934420 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "0294d374c1b04df7b6e25c703c3c9e38"
format_stamp: "Formatted at 2026-08-10 06:24:18 on dist-test-slave-q9h9"
server_key: "768c45853f390b3198e5af38151f34bd"
server_key_iv: "5d5ccb99cb8ba0b189c843a020e1ae8b"
server_key_version: "encryptionkey@0"
I20260810 06:24:18.937752 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:24:18.939947 1755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:18.941021 1723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:18.941136 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "0294d374c1b04df7b6e25c703c3c9e38"
format_stamp: "Formatted at 2026-08-10 06:24:18 on dist-test-slave-q9h9"
server_key: "768c45853f390b3198e5af38151f34bd"
server_key_iv: "5d5ccb99cb8ba0b189c843a020e1ae8b"
server_key_version: "encryptionkey@0"
I20260810 06:24:18.941224 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:18.945336 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:18.947888 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:18.948047 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:18.954468 1723 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46169
I20260810 06:24:18.954510 1831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46169 every 8 connection(s)
I20260810 06:24:18.955320 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:18.956353 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 1723
I20260810 06:24:18.956476 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:18.956735 32563 external_mini_cluster.cc:1428] Setting key 5ca66faf1513211bb2cf85123f351e97
I20260810 06:24:18.958209 1832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:18.961820 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:18.957925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56831 (local address 127.0.0.1:46169)
0810 06:24:18.958356 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:18.958374 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:18.958415 (+ 41us) server_negotiation.cc:407] Connection header received
0810 06:24:18.959206 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:18.959227 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:18.959570 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:18.959910 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:18.960518 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:18.960523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:18.960537 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:18.960549 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:18.960604 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:18.960628 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:18.960767 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:18.960832 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 06:24:18.960974 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":107,"threads_started":1}
I20260810 06:24:18.964334 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:24:18.964439 1832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38: Bootstrap starting.
I20260810 06:24:18.967027 1832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:18.967891 1832 log.cc:824] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:18.968086 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:18.966572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36005 (local address 127.0.0.1:46169)
0810 06:24:18.966674 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:18.966678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:18.966913 (+ 235us) server_negotiation.cc:407] Connection header received
0810 06:24:18.967029 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:18.967031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:18.967058 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:18.967126 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:18.967754 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:18.967756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:18.967758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:18.967767 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:18.967789 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:18.967798 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:18.967856 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:18.967946 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 06:24:18.967974 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 06:24:18.969781 1832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38: No bootstrap required, opened a new log
I20260810 06:24:18.973204 1832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0294d374c1b04df7b6e25c703c3c9e38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } }
I20260810 06:24:18.973428 1832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:18.973500 1832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0294d374c1b04df7b6e25c703c3c9e38, State: Initialized, Role: FOLLOWER
I20260810 06:24:18.973868 1832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0294d374c1b04df7b6e25c703c3c9e38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } }
I20260810 06:24:18.973985 1832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:18.974071 1832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:18.974165 1832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:18.974961 1832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0294d374c1b04df7b6e25c703c3c9e38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } }
I20260810 06:24:18.975252 1832 leader_election.cc:302] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0294d374c1b04df7b6e25c703c3c9e38; no voters:
I20260810 06:24:18.975521 1832 leader_election.cc:288] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:18.975600 1847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:18.975806 1847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38 [term 1 LEADER]: Becoming Leader. State: Replica: 0294d374c1b04df7b6e25c703c3c9e38, State: Running, Role: LEADER
I20260810 06:24:18.976114 1847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0294d374c1b04df7b6e25c703c3c9e38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } }
I20260810 06:24:18.976552 1832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:18.977675 1848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0294d374c1b04df7b6e25c703c3c9e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0294d374c1b04df7b6e25c703c3c9e38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } } }
I20260810 06:24:18.977669 1849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0294d374c1b04df7b6e25c703c3c9e38. Latest consensus state: current_term: 1 leader_uuid: "0294d374c1b04df7b6e25c703c3c9e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0294d374c1b04df7b6e25c703c3c9e38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } } }
I20260810 06:24:18.977813 1848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:18.977823 1849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:18.978654 1858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:18.981060 1858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:18.985487 1858 catalog_manager.cc:1347] Generated new cluster ID: 4ec6a29f611344998a6233b7113f1692
I20260810 06:24:18.985546 1858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:18.991967 1858 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:18.992209 1858 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:18.996212 1858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0294d374c1b04df7b6e25c703c3c9e38: Generated new TSK 0
I20260810 06:24:18.996856 1858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:19.014820 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 1723
2026-08-10T06:24:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1490 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:19 dist-test-slave-q9h9 krb5kdc[1877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:19 dist-test-slave-q9h9 krb5kdc[1877](info): set up 2 sockets
Aug 10 06:24:19 dist-test-slave-q9h9 krb5kdc[1877](info): commencing operation
krb5kdc: starting...
W20260810 06:24:20.193930 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:20 dist-test-slave-q9h9 krb5kdc[1877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:24:20.305853 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:20.411518 1897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:20.411751 1897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:20.411806 1897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:20.415681 1897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:20.415733 1897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:20.415760 1897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:20.415783 1897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:20.415812 1897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:20.420761 1897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:20.421664 1897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:20.422598 1897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:20.430330 1911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:20.430343 1902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:20.430400 1909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:20.430392 1897 server_base.cc:1034] running on GCE node
I20260810 06:24:20.431020 1897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:20.431984 1897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:20.433149 1897 hybrid_clock.cc:648] HybridClock initialized: now 1786343060433136 us; error 23 us; skew 500 ppm
Aug 10 06:24:20 dist-test-slave-q9h9 krb5kdc[1877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:20.435662 1897 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:24:20.436455 1897 webserver.cc:460] Webserver started at http://127.0.0.1:44817/ using document root <none> and password file <none>
I20260810 06:24:20.436937 1897 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:20.437001 1897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:20.437212 1897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:20.439121 1897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "dd80692cb7194a2981226adc202fcc6d"
format_stamp: "Formatted at 2026-08-10 06:24:20 on dist-test-slave-q9h9"
server_key: "1449fff49f2ce4c4ff29f31d7c5a6c8e"
server_key_iv: "76b81faded835ecf9f196a539a215589"
server_key_version: "encryptionkey@0"
I20260810 06:24:20.439648 1897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "dd80692cb7194a2981226adc202fcc6d"
format_stamp: "Formatted at 2026-08-10 06:24:20 on dist-test-slave-q9h9"
server_key: "1449fff49f2ce4c4ff29f31d7c5a6c8e"
server_key_iv: "76b81faded835ecf9f196a539a215589"
server_key_version: "encryptionkey@0"
I20260810 06:24:20.442982 1897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:24:20.445058 1923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:20.446189 1897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:24:20.446295 1897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "dd80692cb7194a2981226adc202fcc6d"
format_stamp: "Formatted at 2026-08-10 06:24:20 on dist-test-slave-q9h9"
server_key: "1449fff49f2ce4c4ff29f31d7c5a6c8e"
server_key_iv: "76b81faded835ecf9f196a539a215589"
server_key_version: "encryptionkey@0"
I20260810 06:24:20.446379 1897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:20.464871 1897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:20.467257 1897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:20.467415 1897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:20.473964 1897 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42271
I20260810 06:24:20.473982 1999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42271 every 8 connection(s)
I20260810 06:24:20.474840 1897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:20.477351 2001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:20.480886 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 1897
I20260810 06:24:20.481025 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:20.481310 32563 external_mini_cluster.cc:1428] Setting key 3e63d5deb506ceeed503d937567046a4
I20260810 06:24:20.482908 2001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d: Bootstrap starting.
I20260810 06:24:20.484954 2001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:20.485644 2001 log.cc:824] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:20.487334 2001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d: No bootstrap required, opened a new log
Aug 10 06:24:20 dist-test-slave-q9h9 krb5kdc[1877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 06:24:20.491065 2001 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd80692cb7194a2981226adc202fcc6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42271 } }
I20260810 06:24:20.491267 2001 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:20.491333 2001 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd80692cb7194a2981226adc202fcc6d, State: Initialized, Role: FOLLOWER
I20260810 06:24:20.491717 2001 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd80692cb7194a2981226adc202fcc6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42271 } }
I20260810 06:24:20.491843 2001 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:20.491907 2001 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:20.491995 2001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:20.492733 2001 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd80692cb7194a2981226adc202fcc6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42271 } }
I20260810 06:24:20.493026 2001 leader_election.cc:302] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd80692cb7194a2981226adc202fcc6d; no voters:
I20260810 06:24:20.493287 2001 leader_election.cc:288] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:20.493364 2006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:20.493633 2006 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d [term 1 LEADER]: Becoming Leader. State: Replica: dd80692cb7194a2981226adc202fcc6d, State: Running, Role: LEADER
I20260810 06:24:20.493969 2006 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd80692cb7194a2981226adc202fcc6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42271 } }
I20260810 06:24:20.494516 2001 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:20.494699 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:20.482626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42207 (local address 127.0.0.1:42271)
0810 06:24:20.483050 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:20.483064 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:20.483099 (+ 35us) server_negotiation.cc:407] Connection header received
0810 06:24:20.483698 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:20.483714 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:20.484018 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:20.484304 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:20.485099 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:20.486112 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:20.486951 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:20.487394 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:20.487526 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:20.490144 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:20.490150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:20.490161 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:20.490188 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:20.491769 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:20.492234 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:20.492239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:20.492243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:20.492350 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:20.492621 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:20.492624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:20.492626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:20.493017 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:20.493210 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:20.493510 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 06:24:20.493804 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":123,"threads_started":1}
I20260810 06:24:20.495044 2006 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd80692cb7194a2981226adc202fcc6d. Latest consensus state: current_term: 1 leader_uuid: "dd80692cb7194a2981226adc202fcc6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd80692cb7194a2981226adc202fcc6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42271 } } }
I20260810 06:24:20.495234 2006 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:20.495435 2011 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd80692cb7194a2981226adc202fcc6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd80692cb7194a2981226adc202fcc6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42271 } } }
I20260810 06:24:20.495573 2011 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:20.495653 2021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:20.497782 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:24:20.498667 2021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:20.504215 2021 catalog_manager.cc:1347] Generated new cluster ID: 8a0ba49a986b48b289daaeab199963aa
I20260810 06:24:20.504276 2021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:20.508060 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:20.500120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50887 (local address 127.0.0.1:42271)
0810 06:24:20.500228 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:20.500232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:20.500484 (+ 252us) server_negotiation.cc:407] Connection header received
0810 06:24:20.500609 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:20.500612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:20.500657 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:20.500733 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:20.501473 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:20.502275 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:20.503174 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:20.503409 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:20.503493 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:20.505068 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:20.505071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:20.505074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:20.505097 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:20.506534 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:20.506955 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:20.506959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:20.506961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:20.507031 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:20.507271 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:20.507273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:20.507275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:20.507442 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:20.507520 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:20.507826 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 06:24:20.507926 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:24:20.519246 2021 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:20.520036 2021 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:20.526778 2021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd80692cb7194a2981226adc202fcc6d: Generated new TSK 0
I20260810 06:24:20.527354 2021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:20.532949 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 1897
2026-08-10T06:24:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1508 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:20 dist-test-slave-q9h9 krb5kdc[2057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:20 dist-test-slave-q9h9 krb5kdc[2057](info): set up 2 sockets
Aug 10 06:24:20 dist-test-slave-q9h9 krb5kdc[2057](info): commencing operation
krb5kdc: starting...
W20260810 06:24:21.703058 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:21 dist-test-slave-q9h9 krb5kdc[2057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:24:21.815989 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:21.923339 2082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:21.923581 2082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:21.923619 2082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:21.927446 2082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:21.927491 2082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:21.927508 2082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:21.927528 2082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:21.927542 2082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:21.932485 2082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:21.933375 2082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:21.934320 2082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:21.942274 2094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:21.942344 2082 server_base.cc:1034] running on GCE node
W20260810 06:24:21.942286 2089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:21.942286 2090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:21.943056 2082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:21.944044 2082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:21.945202 2082 hybrid_clock.cc:648] HybridClock initialized: now 1786343061945188 us; error 27 us; skew 500 ppm
Aug 10 06:24:21 dist-test-slave-q9h9 krb5kdc[2057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:21.947736 2082 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:24:21.948520 2082 webserver.cc:460] Webserver started at http://127.0.0.1:43695/ using document root <none> and password file <none>
I20260810 06:24:21.949013 2082 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:21.949075 2082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:21.949272 2082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:21.951243 2082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "ac6d01fa63af4aa28ea2fe6786e82c0c"
format_stamp: "Formatted at 2026-08-10 06:24:21 on dist-test-slave-q9h9"
server_key: "261ae5e8aca7e25df738a79cfb9c0f94"
server_key_iv: "22bdb89dcbe042bc6592ad42def02eaf"
server_key_version: "encryptionkey@0"
I20260810 06:24:21.951764 2082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "ac6d01fa63af4aa28ea2fe6786e82c0c"
format_stamp: "Formatted at 2026-08-10 06:24:21 on dist-test-slave-q9h9"
server_key: "261ae5e8aca7e25df738a79cfb9c0f94"
server_key_iv: "22bdb89dcbe042bc6592ad42def02eaf"
server_key_version: "encryptionkey@0"
I20260810 06:24:21.955169 2082 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:24:21.957379 2107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:21.958541 2082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:24:21.958647 2082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "ac6d01fa63af4aa28ea2fe6786e82c0c"
format_stamp: "Formatted at 2026-08-10 06:24:21 on dist-test-slave-q9h9"
server_key: "261ae5e8aca7e25df738a79cfb9c0f94"
server_key_iv: "22bdb89dcbe042bc6592ad42def02eaf"
server_key_version: "encryptionkey@0"
I20260810 06:24:21.958729 2082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:21.976400 2082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:21.978857 2082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:21.979022 2082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:21.985556 2082 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37131
I20260810 06:24:21.985607 2197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37131 every 8 connection(s)
I20260810 06:24:21.986469 2082 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:21.989040 2198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:21.991092 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 2082
I20260810 06:24:21.991215 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:21.991461 32563 external_mini_cluster.cc:1428] Setting key 0c30cfc2868dc877dd128db6d1b625be
I20260810 06:24:21.995289 2198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c: Bootstrap starting.
I20260810 06:24:21.997815 2198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:21.998713 2198 log.cc:824] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:22.000653 2198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c: No bootstrap required, opened a new log
I20260810 06:24:22.000949 2201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:21.992734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33347 (local address 127.0.0.1:37131)
0810 06:24:21.993161 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:21.993182 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:21.993225 (+ 43us) server_negotiation.cc:407] Connection header received
0810 06:24:21.994159 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:21.994183 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:21.994599 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:21.994974 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:21.995362 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:21.996799 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:21.997686 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:21.998403 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:21.998563 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:21.999293 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:21.999299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:21.999315 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:21.999715 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:21.999770 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:21.999794 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:21.999869 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:22.000020 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 06:24:22.000202 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":103,"threads_started":1}
I20260810 06:24:22.003536 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:24:22.004226 2198 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6d01fa63af4aa28ea2fe6786e82c0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37131 } }
I20260810 06:24:22.004499 2198 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:22.004558 2198 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6d01fa63af4aa28ea2fe6786e82c0c, State: Initialized, Role: FOLLOWER
I20260810 06:24:22.005012 2198 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6d01fa63af4aa28ea2fe6786e82c0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37131 } }
I20260810 06:24:22.005168 2198 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:22.005235 2198 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:22.005340 2198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:22.006389 2198 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6d01fa63af4aa28ea2fe6786e82c0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37131 } }
I20260810 06:24:22.006697 2198 leader_election.cc:302] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac6d01fa63af4aa28ea2fe6786e82c0c; no voters:
I20260810 06:24:22.006981 2198 leader_election.cc:288] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:22.007071 2215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:22.007315 2215 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c [term 1 LEADER]: Becoming Leader. State: Replica: ac6d01fa63af4aa28ea2fe6786e82c0c, State: Running, Role: LEADER
I20260810 06:24:22.007673 2215 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6d01fa63af4aa28ea2fe6786e82c0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37131 } }
I20260810 06:24:22.008167 2198 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:22.009596 2220 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac6d01fa63af4aa28ea2fe6786e82c0c. Latest consensus state: current_term: 1 leader_uuid: "ac6d01fa63af4aa28ea2fe6786e82c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6d01fa63af4aa28ea2fe6786e82c0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37131 } } }
I20260810 06:24:22.009737 2220 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:22.010289 2201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:22.005699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46945 (local address 127.0.0.1:37131)
0810 06:24:22.005811 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:22.005815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:22.005980 (+ 165us) server_negotiation.cc:407] Connection header received
0810 06:24:22.006125 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:22.006127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:22.006164 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:22.006223 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:22.006504 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:22.007107 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:22.008370 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:22.008678 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:22.008777 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:22.009715 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:22.009718 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:22.009721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:22.009733 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:22.009760 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:22.009773 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:22.009839 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:22.010100 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 06:24:22.010150 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":41}
I20260810 06:24:22.009171 2219 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac6d01fa63af4aa28ea2fe6786e82c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6d01fa63af4aa28ea2fe6786e82c0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37131 } } }
I20260810 06:24:22.010675 2237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:22.010681 2219 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:22.013749 2237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:22.019701 2237 catalog_manager.cc:1347] Generated new cluster ID: 8ce67c0b77814753a1413412d387a004
I20260810 06:24:22.019773 2237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:22.034744 2237 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:22.035959 2237 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:22.042304 2237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac6d01fa63af4aa28ea2fe6786e82c0c: Generated new TSK 0
I20260810 06:24:22.043020 2237 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:22.048583 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 2082
2026-08-10T06:24:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1519 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:22 dist-test-slave-q9h9 krb5kdc[2249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:22 dist-test-slave-q9h9 krb5kdc[2249](info): set up 2 sockets
Aug 10 06:24:22 dist-test-slave-q9h9 krb5kdc[2249](info): commencing operation
krb5kdc: starting...
W20260810 06:24:23.212318 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:23 dist-test-slave-q9h9 krb5kdc[2249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:24:23.326529 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42393
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:23.433849 2270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:23.434146 2270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:23.434212 2270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:23.438071 2270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:23.438127 2270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:23.438153 2270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:23.438174 2270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:23.438201 2270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:23.443228 2270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:23.444182 2270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:23.445168 2270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:23.453038 2283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:23.453076 2277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:23.453136 2270 server_base.cc:1034] running on GCE node
W20260810 06:24:23.453076 2279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:23.453833 2270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:23.454890 2270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:23.456055 2270 hybrid_clock.cc:648] HybridClock initialized: now 1786343063456032 us; error 33 us; skew 500 ppm
Aug 10 06:24:23 dist-test-slave-q9h9 krb5kdc[2249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:23.458729 2270 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:24:23.459609 2270 webserver.cc:460] Webserver started at http://127.0.0.1:39141/ using document root <none> and password file <none>
I20260810 06:24:23.460132 2270 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:23.460194 2270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:23.460387 2270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:23.462417 2270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "4152756bec3449fd9e4c234228487c69"
format_stamp: "Formatted at 2026-08-10 06:24:23 on dist-test-slave-q9h9"
server_key: "e7c484a3cd4d1cbcde927cbbc828171e"
server_key_iv: "ee34da12bea66d1fbb4287efd537bc02"
server_key_version: "encryptionkey@0"
I20260810 06:24:23.462954 2270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "4152756bec3449fd9e4c234228487c69"
format_stamp: "Formatted at 2026-08-10 06:24:23 on dist-test-slave-q9h9"
server_key: "e7c484a3cd4d1cbcde927cbbc828171e"
server_key_iv: "ee34da12bea66d1fbb4287efd537bc02"
server_key_version: "encryptionkey@0"
I20260810 06:24:23.466440 2270 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:24:23.468752 2293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:23.469916 2270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:24:23.470057 2270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "4152756bec3449fd9e4c234228487c69"
format_stamp: "Formatted at 2026-08-10 06:24:23 on dist-test-slave-q9h9"
server_key: "e7c484a3cd4d1cbcde927cbbc828171e"
server_key_iv: "ee34da12bea66d1fbb4287efd537bc02"
server_key_version: "encryptionkey@0"
I20260810 06:24:23.470155 2270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:23.474435 2270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:23.477155 2270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:23.477311 2270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:23.483906 2270 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42393
I20260810 06:24:23.483918 2385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42393 every 8 connection(s)
I20260810 06:24:23.484759 2270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:23.487401 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:23.491423 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 2270
I20260810 06:24:23.491554 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:23.491847 32563 external_mini_cluster.cc:1428] Setting key cdeeae89e7673696f4b85691e2023d34
I20260810 06:24:23.493006 2386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69: Bootstrap starting.
I20260810 06:24:23.495366 2386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:23.496186 2386 log.cc:824] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:23.496788 2390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:23.493149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39767 (local address 127.0.0.1:42393)
0810 06:24:23.493541 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:23.493555 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:23.493589 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:24:23.494251 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:23.494268 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:23.494535 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:23.494820 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:23.495614 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:23.495619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:23.495630 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:23.495647 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:23.495704 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:23.495722 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:23.495804 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:23.495927 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 06:24:23.496064 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":102,"threads_started":1}
I20260810 06:24:23.498153 2386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69: No bootstrap required, opened a new log
I20260810 06:24:23.499349 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:24:23.501888 2386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4152756bec3449fd9e4c234228487c69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } }
I20260810 06:24:23.502170 2386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:23.502231 2386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4152756bec3449fd9e4c234228487c69, State: Initialized, Role: FOLLOWER
I20260810 06:24:23.502671 2390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:23.501207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39873 (local address 127.0.0.1:42393)
0810 06:24:23.501323 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:23.501327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:23.501491 (+ 164us) server_negotiation.cc:407] Connection header received
0810 06:24:23.501567 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:23.501569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:23.501601 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:23.501677 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:23.502322 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:23.502325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:23.502327 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:23.502339 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:23.502365 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:23.502377 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:23.502462 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:23.502532 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 06:24:23.502567 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:24:23.502679 2386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4152756bec3449fd9e4c234228487c69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } }
I20260810 06:24:23.502854 2386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:23.502915 2386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:23.503005 2386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:23.503814 2386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4152756bec3449fd9e4c234228487c69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } }
I20260810 06:24:23.504115 2386 leader_election.cc:302] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4152756bec3449fd9e4c234228487c69; no voters:
I20260810 06:24:23.504398 2386 leader_election.cc:288] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:23.504474 2403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:23.504683 2403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69 [term 1 LEADER]: Becoming Leader. State: Replica: 4152756bec3449fd9e4c234228487c69, State: Running, Role: LEADER
I20260810 06:24:23.505014 2403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4152756bec3449fd9e4c234228487c69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } }
I20260810 06:24:23.505460 2386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:23.506644 2407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4152756bec3449fd9e4c234228487c69. Latest consensus state: current_term: 1 leader_uuid: "4152756bec3449fd9e4c234228487c69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4152756bec3449fd9e4c234228487c69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } } }
I20260810 06:24:23.506655 2406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4152756bec3449fd9e4c234228487c69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4152756bec3449fd9e4c234228487c69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } } }
I20260810 06:24:23.506835 2407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:23.506855 2406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:23.507743 2424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:23.510350 2424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:23.515424 2424 catalog_manager.cc:1347] Generated new cluster ID: 5c87f88d8ecf4e5a8ac519ea7d7598a5
I20260810 06:24:23.515484 2424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:23.525398 2424 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:23.525657 2424 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:23.529734 2424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4152756bec3449fd9e4c234228487c69: Generated new TSK 0
I20260810 06:24:23.530442 2424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:23.538682 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 2270
2026-08-10T06:24:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1481 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:23 dist-test-slave-q9h9 krb5kdc[2443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:23 dist-test-slave-q9h9 krb5kdc[2443](info): set up 2 sockets
Aug 10 06:24:23 dist-test-slave-q9h9 krb5kdc[2443](info): commencing operation
krb5kdc: starting...
W20260810 06:24:24.687222 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:24 dist-test-slave-q9h9 krb5kdc[2443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:24:24.797600 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:24.903441 2462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:24.903664 2462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:24.903704 2462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:24.907603 2462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:24.907652 2462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:24.907673 2462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:24.907706 2462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:24.907732 2462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:24.912708 2462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:24.913609 2462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:24.914579 2462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:24.922405 2474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:24.922565 2473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:24.922509 2462 server_base.cc:1034] running on GCE node
W20260810 06:24:24.922511 2477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:24.923206 2462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:24.924167 2462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:24.925323 2462 hybrid_clock.cc:648] HybridClock initialized: now 1786343064925305 us; error 29 us; skew 500 ppm
Aug 10 06:24:24 dist-test-slave-q9h9 krb5kdc[2443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:24.932189 2462 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:24:24.933012 2462 webserver.cc:460] Webserver started at http://127.0.0.1:34899/ using document root <none> and password file <none>
I20260810 06:24:24.933507 2462 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:24.933568 2462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:24.933765 2462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:24.935714 2462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "93921e25cdda4fbc96808b91d1d48bd5"
format_stamp: "Formatted at 2026-08-10 06:24:24 on dist-test-slave-q9h9"
server_key: "d00002d6b24e9f48ec79e73a5a31b8ac"
server_key_iv: "f778464b74421e28bb078bd6c9fa3e94"
server_key_version: "encryptionkey@0"
I20260810 06:24:24.936244 2462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "93921e25cdda4fbc96808b91d1d48bd5"
format_stamp: "Formatted at 2026-08-10 06:24:24 on dist-test-slave-q9h9"
server_key: "d00002d6b24e9f48ec79e73a5a31b8ac"
server_key_iv: "f778464b74421e28bb078bd6c9fa3e94"
server_key_version: "encryptionkey@0"
I20260810 06:24:24.939718 2462 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 06:24:24.941862 2487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:24.943004 2462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:24.943118 2462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "93921e25cdda4fbc96808b91d1d48bd5"
format_stamp: "Formatted at 2026-08-10 06:24:24 on dist-test-slave-q9h9"
server_key: "d00002d6b24e9f48ec79e73a5a31b8ac"
server_key_iv: "f778464b74421e28bb078bd6c9fa3e94"
server_key_version: "encryptionkey@0"
I20260810 06:24:24.943210 2462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:24.962316 2462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:24.965152 2462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:24.965317 2462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:24.971968 2462 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41691
I20260810 06:24:24.971997 2577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41691 every 8 connection(s)
I20260810 06:24:24.972851 2462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:24.975425 2578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:24.980520 2578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5: Bootstrap starting.
I20260810 06:24:24.982645 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 2462
I20260810 06:24:24.982764 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:24.982899 2578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:24.983034 32563 external_mini_cluster.cc:1428] Setting key fa2a28fc9864b562c653cd10701b9286
I20260810 06:24:24.983673 2578 log.cc:824] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:24.985776 2578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5: No bootstrap required, opened a new log
I20260810 06:24:24.989794 2578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93921e25cdda4fbc96808b91d1d48bd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41691 } }
I20260810 06:24:24.990060 2578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:24.990165 2578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93921e25cdda4fbc96808b91d1d48bd5, State: Initialized, Role: FOLLOWER
I20260810 06:24:24.990587 2578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93921e25cdda4fbc96808b91d1d48bd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41691 } }
I20260810 06:24:24.990708 2578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:24.990763 2578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:24.990855 2578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:24.991624 2578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93921e25cdda4fbc96808b91d1d48bd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41691 } }
Aug 10 06:24:24 dist-test-slave-q9h9 krb5kdc[2443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 06:24:24.991919 2578 leader_election.cc:302] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93921e25cdda4fbc96808b91d1d48bd5; no voters:
I20260810 06:24:24.992223 2578 leader_election.cc:288] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:24.992295 2584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:24.992501 2584 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5 [term 1 LEADER]: Becoming Leader. State: Replica: 93921e25cdda4fbc96808b91d1d48bd5, State: Running, Role: LEADER
I20260810 06:24:24.992802 2584 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93921e25cdda4fbc96808b91d1d48bd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41691 } }
I20260810 06:24:24.993445 2578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:24.993963 2586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93921e25cdda4fbc96808b91d1d48bd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93921e25cdda4fbc96808b91d1d48bd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41691 } } }
I20260810 06:24:24.994159 2586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:24.994589 2595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:24.996044 2588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93921e25cdda4fbc96808b91d1d48bd5. Latest consensus state: current_term: 1 leader_uuid: "93921e25cdda4fbc96808b91d1d48bd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93921e25cdda4fbc96808b91d1d48bd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41691 } } }
I20260810 06:24:24.996150 2588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:24.997704 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:24.984208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:59991 (local address 127.0.0.1:41691)
0810 06:24:24.984643 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:24.984661 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:24.984701 (+ 40us) server_negotiation.cc:407] Connection header received
0810 06:24:24.985499 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:24.985520 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:24.985924 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:24.986290 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:24.986949 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:24.988400 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:24.989202 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:24.989872 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:24.990062 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:24.992562 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:24.992568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:24.992585 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:24.992617 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:24.994439 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:24.995082 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:24.995088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:24.995093 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:24.995231 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:24.995638 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:24.995642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:24.995644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:24.996109 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:24.996337 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:24.996800 (+ 463us) server_negotiation.cc:299] Negotiation successful
0810 06:24:24.997027 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":116,"threads_started":1}
I20260810 06:24:24.998482 2595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:25.000759 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:24:25.003847 2595 catalog_manager.cc:1347] Generated new cluster ID: 14b02b1bdb5a43499c571025addf7e16
I20260810 06:24:25.003904 2595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:25.011649 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:25.003237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:34811 (local address 127.0.0.1:41691)
0810 06:24:25.003347 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:25.003351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:25.003364 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:24:25.003394 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:25.003397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:25.003437 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:25.003516 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:25.004395 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:25.005164 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:25.006365 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:25.006714 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:25.006803 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:25.008363 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:25.008366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:25.008369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:25.008395 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:25.009961 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:25.010474 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:25.010477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:25.010479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:25.010563 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:25.010837 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:25.010840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:25.010842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:25.011036 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:25.011185 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:25.011393 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 06:24:25.011483 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:24:25.014760 2595 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:25.015540 2595 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:25.024618 2595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93921e25cdda4fbc96808b91d1d48bd5: Generated new TSK 0
I20260810 06:24:25.025184 2595 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:25.032371 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 2462
W20260810 06:24:25.044319 2440 connection.cc:535] client connection to 127.0.0.1:41691 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41691): Connection reset by peer (error 104)
2026-08-10T06:24:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1496 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:25 dist-test-slave-q9h9 krb5kdc[2628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:25 dist-test-slave-q9h9 krb5kdc[2628](info): set up 2 sockets
Aug 10 06:24:25 dist-test-slave-q9h9 krb5kdc[2628](info): commencing operation
krb5kdc: starting...
W20260810 06:24:26.187929 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:26 dist-test-slave-q9h9 krb5kdc[2628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:24:26.297978 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40803
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:26.401736 2643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:26.401961 2643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:26.402034 2643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:26.405836 2643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:26.405886 2643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:26.405912 2643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:26.405933 2643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:26.405961 2643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:26.410952 2643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:26.411823 2643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:26.412691 2643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:26.420680 2656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:26.420693 2655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:26.420698 2661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:26.420948 2643 server_base.cc:1034] running on GCE node
I20260810 06:24:26.421442 2643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:26.422474 2643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:26.423625 2643 hybrid_clock.cc:648] HybridClock initialized: now 1786343066423602 us; error 36 us; skew 500 ppm
Aug 10 06:24:26 dist-test-slave-q9h9 krb5kdc[2628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:26.426116 2643 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:24:26.426939 2643 webserver.cc:460] Webserver started at http://127.0.0.1:37607/ using document root <none> and password file <none>
I20260810 06:24:26.427395 2643 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:26.427449 2643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:26.427592 2643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:26.429373 2643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "e3069269d711418e9e7edb536d0acb89"
format_stamp: "Formatted at 2026-08-10 06:24:26 on dist-test-slave-q9h9"
server_key: "f63ef2d9263ce96147d6a4afce352e76"
server_key_iv: "4356394512813c33776f2a7f6426dcfa"
server_key_version: "encryptionkey@0"
I20260810 06:24:26.429838 2643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "e3069269d711418e9e7edb536d0acb89"
format_stamp: "Formatted at 2026-08-10 06:24:26 on dist-test-slave-q9h9"
server_key: "f63ef2d9263ce96147d6a4afce352e76"
server_key_iv: "4356394512813c33776f2a7f6426dcfa"
server_key_version: "encryptionkey@0"
I20260810 06:24:26.433080 2643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:24:26.435241 2671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:26.436316 2643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:24:26.436411 2643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "e3069269d711418e9e7edb536d0acb89"
format_stamp: "Formatted at 2026-08-10 06:24:26 on dist-test-slave-q9h9"
server_key: "f63ef2d9263ce96147d6a4afce352e76"
server_key_iv: "4356394512813c33776f2a7f6426dcfa"
server_key_version: "encryptionkey@0"
I20260810 06:24:26.436498 2643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:26.459992 2643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:26.462392 2643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:26.462545 2643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:26.468906 2643 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40803
I20260810 06:24:26.468922 2746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40803 every 8 connection(s)
I20260810 06:24:26.469719 2643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:26.472289 2749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:26.472774 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 2643
I20260810 06:24:26.472914 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:26.473214 32563 external_mini_cluster.cc:1428] Setting key dc14d8f30c16c34b6dfc8e85e41f045c
I20260810 06:24:26.478820 2749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89: Bootstrap starting.
I20260810 06:24:26.481195 2749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:26.481463 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:26.474423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:43195 (local address 127.0.0.1:40803)
0810 06:24:26.474792 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:26.474817 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:26.474849 (+ 32us) server_negotiation.cc:407] Connection header received
0810 06:24:26.475486 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:26.475503 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:26.475804 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:26.476055 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:26.476413 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:26.477351 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:26.478573 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:26.479021 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:26.479130 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:26.479936 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:26.479940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:26.479951 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:26.480291 (+ 340us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:26.480341 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:26.480361 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:26.480456 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:26.480639 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 06:24:26.480756 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":104,"threads_started":1}
I20260810 06:24:26.481972 2749 log.cc:824] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:26.483916 2749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89: No bootstrap required, opened a new log
I20260810 06:24:26.484161 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:24:26.487453 2749 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3069269d711418e9e7edb536d0acb89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40803 } }
I20260810 06:24:26.487648 2749 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:26.487685 2749 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3069269d711418e9e7edb536d0acb89, State: Initialized, Role: FOLLOWER
I20260810 06:24:26.488075 2749 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3069269d711418e9e7edb536d0acb89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40803 } }
I20260810 06:24:26.488205 2749 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:26.488266 2749 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:26.488351 2749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:26.489254 2749 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3069269d711418e9e7edb536d0acb89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40803 } }
I20260810 06:24:26.489522 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:26.485928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:46519 (local address 127.0.0.1:40803)
0810 06:24:26.486042 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:26.486045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:26.486217 (+ 172us) server_negotiation.cc:407] Connection header received
0810 06:24:26.486328 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:26.486331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:26.486370 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:26.486430 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:26.486678 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:26.487167 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:26.488041 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:26.488319 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:26.488410 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:26.489108 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:26.489111 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:26.489114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:26.489126 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:26.489148 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:26.489157 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:26.489220 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:26.489372 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 06:24:26.489403 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 06:24:26.489553 2749 leader_election.cc:302] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3069269d711418e9e7edb536d0acb89; no voters:
I20260810 06:24:26.489845 2749 leader_election.cc:288] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:26.489928 2769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:26.490168 2769 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89 [term 1 LEADER]: Becoming Leader. State: Replica: e3069269d711418e9e7edb536d0acb89, State: Running, Role: LEADER
I20260810 06:24:26.490507 2769 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3069269d711418e9e7edb536d0acb89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40803 } }
I20260810 06:24:26.491065 2749 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:26.491641 2769 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3069269d711418e9e7edb536d0acb89. Latest consensus state: current_term: 1 leader_uuid: "e3069269d711418e9e7edb536d0acb89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3069269d711418e9e7edb536d0acb89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40803 } } }
I20260810 06:24:26.491847 2769 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:26.492025 2770 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3069269d711418e9e7edb536d0acb89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3069269d711418e9e7edb536d0acb89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40803 } } }
I20260810 06:24:26.492129 2770 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:26.492370 2782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:26.495380 2782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:26.500183 2782 catalog_manager.cc:1347] Generated new cluster ID: 28a4531ebd444969bfc7614f76f8871f
I20260810 06:24:26.500284 2782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:26.506129 2782 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:26.506894 2782 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:26.512537 2782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3069269d711418e9e7edb536d0acb89: Generated new TSK 0
I20260810 06:24:26.513064 2782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:26.517139 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 2643
W20260810 06:24:26.530609 2624 connection.cc:535] client connection to 127.0.0.1:40803 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40803): Connection reset by peer (error 104)
2026-08-10T06:24:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1486 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:26 dist-test-slave-q9h9 krb5kdc[2809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:26 dist-test-slave-q9h9 krb5kdc[2809](info): set up 2 sockets
Aug 10 06:24:26 dist-test-slave-q9h9 krb5kdc[2809](info): commencing operation
krb5kdc: starting...
W20260810 06:24:27.663092 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:27 dist-test-slave-q9h9 krb5kdc[2809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:24:27.772861 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:27.876564 2827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:27.876786 2827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:27.876842 2827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:27.880692 2827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:27.880743 2827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:27.880770 2827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:27.880791 2827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:27.880820 2827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:27.885655 2827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:27.886556 2827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:27.887431 2827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:24:27.895110 2827 server_base.cc:1034] running on GCE node
W20260810 06:24:27.895121 2838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:27.895115 2833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:27.895074 2835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:27.895812 2827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:27.896761 2827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:27.897914 2827 hybrid_clock.cc:648] HybridClock initialized: now 1786343067897888 us; error 36 us; skew 500 ppm
Aug 10 06:24:27 dist-test-slave-q9h9 krb5kdc[2809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343067, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:27.900596 2827 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:24:27.901382 2827 webserver.cc:460] Webserver started at http://127.0.0.1:37759/ using document root <none> and password file <none>
I20260810 06:24:27.901872 2827 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:27.901937 2827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:27.902165 2827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:27.904016 2827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "5a0b2c5d80ce4275b1db0f18991d3b1f"
format_stamp: "Formatted at 2026-08-10 06:24:27 on dist-test-slave-q9h9"
server_key: "894bc7b301a76773b4eb41822f4bb12c"
server_key_iv: "b400a5416ae422cd3b630d095af90003"
server_key_version: "encryptionkey@0"
I20260810 06:24:27.904528 2827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "5a0b2c5d80ce4275b1db0f18991d3b1f"
format_stamp: "Formatted at 2026-08-10 06:24:27 on dist-test-slave-q9h9"
server_key: "894bc7b301a76773b4eb41822f4bb12c"
server_key_iv: "b400a5416ae422cd3b630d095af90003"
server_key_version: "encryptionkey@0"
I20260810 06:24:27.907869 2827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:24:27.909921 2849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:27.910985 2827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:27.911113 2827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "5a0b2c5d80ce4275b1db0f18991d3b1f"
format_stamp: "Formatted at 2026-08-10 06:24:27 on dist-test-slave-q9h9"
server_key: "894bc7b301a76773b4eb41822f4bb12c"
server_key_iv: "b400a5416ae422cd3b630d095af90003"
server_key_version: "encryptionkey@0"
I20260810 06:24:27.911203 2827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:27.915306 2827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:27.918089 2827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:27.918252 2827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:27.924404 2827 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33057
I20260810 06:24:27.924453 2933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33057 every 8 connection(s)
I20260810 06:24:27.925201 2827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:27.927529 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 2827
I20260810 06:24:27.927660 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:27.927713 2935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:27.927933 32563 external_mini_cluster.cc:1428] Setting key a361ed992b8d4d599ec16ba805619b06
I20260810 06:24:27.932624 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:27.929112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:44233 (local address 127.0.0.1:33057)
0810 06:24:27.929509 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:27.929523 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:27.929556 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:24:27.930168 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:27.930186 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:27.930448 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:27.930709 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:27.931487 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:27.931492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:27.931503 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:27.931514 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:27.931564 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:27.931583 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:27.931692 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:27.931791 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 06:24:27.931924 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":122,"threads_started":1}
I20260810 06:24:27.934242 2935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f: Bootstrap starting.
I20260810 06:24:27.935326 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:24:27.936451 2935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:27.937134 2935 log.cc:824] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:27.938689 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:27.937236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:51523 (local address 127.0.0.1:33057)
0810 06:24:27.937341 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:27.937345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:27.937529 (+ 184us) server_negotiation.cc:407] Connection header received
0810 06:24:27.937639 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:27.937641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:27.937671 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:27.937753 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:27.938352 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:27.938354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:27.938356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:27.938366 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:27.938389 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:27.938399 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:27.938463 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:27.938547 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 06:24:27.938577 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 06:24:27.939342 2935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f: No bootstrap required, opened a new log
I20260810 06:24:27.942691 2935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0b2c5d80ce4275b1db0f18991d3b1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } }
I20260810 06:24:27.942894 2935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:27.942938 2935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0b2c5d80ce4275b1db0f18991d3b1f, State: Initialized, Role: FOLLOWER
I20260810 06:24:27.943301 2935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0b2c5d80ce4275b1db0f18991d3b1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } }
I20260810 06:24:27.943413 2935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:27.943473 2935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:27.943562 2935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:27.944343 2935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0b2c5d80ce4275b1db0f18991d3b1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } }
I20260810 06:24:27.944638 2935 leader_election.cc:302] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a0b2c5d80ce4275b1db0f18991d3b1f; no voters:
I20260810 06:24:27.944902 2935 leader_election.cc:288] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:27.945015 2954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:27.945181 2954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f [term 1 LEADER]: Becoming Leader. State: Replica: 5a0b2c5d80ce4275b1db0f18991d3b1f, State: Running, Role: LEADER
I20260810 06:24:27.945516 2954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0b2c5d80ce4275b1db0f18991d3b1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } }
I20260810 06:24:27.946169 2935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:27.947230 2957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a0b2c5d80ce4275b1db0f18991d3b1f. Latest consensus state: current_term: 1 leader_uuid: "5a0b2c5d80ce4275b1db0f18991d3b1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0b2c5d80ce4275b1db0f18991d3b1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } } }
I20260810 06:24:27.947250 2956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a0b2c5d80ce4275b1db0f18991d3b1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0b2c5d80ce4275b1db0f18991d3b1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } } }
I20260810 06:24:27.947359 2957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:27.947407 2956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:27.947753 2974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:27.950533 2974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:27.955252 2974 catalog_manager.cc:1347] Generated new cluster ID: f8ccb81763f44dd9ab3ffcb72f981fb3
I20260810 06:24:27.955313 2974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:27.962976 2974 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:27.963267 2974 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:27.968035 2974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a0b2c5d80ce4275b1db0f18991d3b1f: Generated new TSK 0
I20260810 06:24:27.968710 2974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:27.973713 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 2827
2026-08-10T06:24:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1461 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:28 dist-test-slave-q9h9 krb5kdc[2994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:28 dist-test-slave-q9h9 krb5kdc[2994](info): set up 2 sockets
Aug 10 06:24:28 dist-test-slave-q9h9 krb5kdc[2994](info): commencing operation
krb5kdc: starting...
W20260810 06:24:29.131934 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:29 dist-test-slave-q9h9 krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:24:29.247280 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40659
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:29.354763 3013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:29.355053 3013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:29.355115 3013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:29.359010 3013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:29.359059 3013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:29.359076 3013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:29.359091 3013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:29.359108 3013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:29.364107 3013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40659
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:29.365010 3013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:29.365935 3013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:29.373844 3020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:29.373888 3013 server_base.cc:1034] running on GCE node
W20260810 06:24:29.373853 3023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:29.373862 3027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:29.374619 3013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:29.375590 3013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:29.376734 3013 hybrid_clock.cc:648] HybridClock initialized: now 1786343069376714 us; error 34 us; skew 500 ppm
Aug 10 06:24:29 dist-test-slave-q9h9 krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:29.379288 3013 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:24:29.380102 3013 webserver.cc:460] Webserver started at http://127.0.0.1:43297/ using document root <none> and password file <none>
I20260810 06:24:29.380584 3013 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:29.380648 3013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:29.380849 3013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:29.382810 3013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "f961c8be284c48eda2e2f50f3c2e444b"
format_stamp: "Formatted at 2026-08-10 06:24:29 on dist-test-slave-q9h9"
server_key: "d11ffc2c49ac10071106cfb0f9359c7f"
server_key_iv: "7e2ad650b8696a3a7934cf62defbd1b9"
server_key_version: "encryptionkey@0"
I20260810 06:24:29.383316 3013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "f961c8be284c48eda2e2f50f3c2e444b"
format_stamp: "Formatted at 2026-08-10 06:24:29 on dist-test-slave-q9h9"
server_key: "d11ffc2c49ac10071106cfb0f9359c7f"
server_key_iv: "7e2ad650b8696a3a7934cf62defbd1b9"
server_key_version: "encryptionkey@0"
I20260810 06:24:29.386742 3013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:24:29.388923 3039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:29.390035 3013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:29.390151 3013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "f961c8be284c48eda2e2f50f3c2e444b"
format_stamp: "Formatted at 2026-08-10 06:24:29 on dist-test-slave-q9h9"
server_key: "d11ffc2c49ac10071106cfb0f9359c7f"
server_key_iv: "7e2ad650b8696a3a7934cf62defbd1b9"
server_key_version: "encryptionkey@0"
I20260810 06:24:29.390237 3013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:29.405798 3013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:29.408762 3013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:29.408932 3013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:29.415649 3013 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40659
I20260810 06:24:29.415668 3128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40659 every 8 connection(s)
I20260810 06:24:29.416539 3013 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:29.419255 3129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:29.423095 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 3013
I20260810 06:24:29.423237 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:29.423519 32563 external_mini_cluster.cc:1428] Setting key fb35d60663863a2d3b2ce59ad31fb655
I20260810 06:24:29.425009 3129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b: Bootstrap starting.
I20260810 06:24:29.427253 3129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:29.427954 3129 log.cc:824] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:29.429718 3129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b: No bootstrap required, opened a new log
Aug 10 06:24:29 dist-test-slave-q9h9 krb5kdc[2994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 06:24:29.433319 3129 raft_consensus.cc:348] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f961c8be284c48eda2e2f50f3c2e444b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40659 } }
I20260810 06:24:29.433530 3129 raft_consensus.cc:374] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:29.433565 3129 raft_consensus.cc:729] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f961c8be284c48eda2e2f50f3c2e444b, State: Initialized, Role: FOLLOWER
I20260810 06:24:29.433971 3129 consensus_queue.cc:260] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f961c8be284c48eda2e2f50f3c2e444b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40659 } }
I20260810 06:24:29.434132 3129 raft_consensus.cc:388] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:29.434203 3129 raft_consensus.cc:482] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:29.434303 3129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:29.435107 3129 raft_consensus.cc:504] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f961c8be284c48eda2e2f50f3c2e444b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40659 } }
I20260810 06:24:29.435421 3129 leader_election.cc:302] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f961c8be284c48eda2e2f50f3c2e444b; no voters:
I20260810 06:24:29.435705 3129 leader_election.cc:288] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:29.435791 3138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:29.435999 3138 raft_consensus.cc:686] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b [term 1 LEADER]: Becoming Leader. State: Replica: f961c8be284c48eda2e2f50f3c2e444b, State: Running, Role: LEADER
I20260810 06:24:29.436324 3138 consensus_queue.cc:237] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f961c8be284c48eda2e2f50f3c2e444b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40659 } }
I20260810 06:24:29.436923 3134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:29.424867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:33233 (local address 127.0.0.1:40659)
0810 06:24:29.425287 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:29.425315 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:29.425349 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:24:29.425968 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:29.425983 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:29.426344 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:29.426637 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:29.427443 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:29.428416 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:29.429258 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:29.429712 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:29.429852 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:29.432395 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:29.432400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:29.432411 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:29.432437 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:29.433964 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:29.434499 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:29.434504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:29.434508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:29.434615 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:29.434958 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:29.434960 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:29.434961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:29.435352 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:29.435556 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:29.435870 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 06:24:29.436096 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":136,"threads_started":1}
I20260810 06:24:29.437003 3129 sys_catalog.cc:564] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:29.437969 3140 sys_catalog.cc:455] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f961c8be284c48eda2e2f50f3c2e444b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f961c8be284c48eda2e2f50f3c2e444b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40659 } } }
I20260810 06:24:29.438112 3140 sys_catalog.cc:458] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:29.438465 3139 sys_catalog.cc:455] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f961c8be284c48eda2e2f50f3c2e444b. Latest consensus state: current_term: 1 leader_uuid: "f961c8be284c48eda2e2f50f3c2e444b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f961c8be284c48eda2e2f50f3c2e444b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40659 } } }
I20260810 06:24:29.438513 3150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:29.438567 3139 sys_catalog.cc:458] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:29.440068 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:24:29.441637 3150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:29.447745 3150 catalog_manager.cc:1347] Generated new cluster ID: f693124a9e6b4cc883e945a7f39de668
I20260810 06:24:29.447800 3150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:29.450431 3134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:29.442259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:56067 (local address 127.0.0.1:40659)
0810 06:24:29.442402 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:29.442406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:29.442765 (+ 359us) server_negotiation.cc:407] Connection header received
0810 06:24:29.442862 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:29.442865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:29.442904 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:29.442962 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:29.443561 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:29.444233 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:29.445336 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:29.445658 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:29.445726 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:29.447130 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:29.447133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:29.447135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:29.447157 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:29.448608 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:29.449183 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:29.449187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:29.449188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:29.449267 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:29.449586 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:29.449589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:29.449590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:29.449763 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:29.449875 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:29.450215 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 06:24:29.450298 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 06:24:29.459371 3150 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:29.460152 3150 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:29.466835 3150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f961c8be284c48eda2e2f50f3c2e444b: Generated new TSK 0
I20260810 06:24:29.467533 3150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:29.476604 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 3013
W20260810 06:24:29.488880 2989 connection.cc:535] client connection to 127.0.0.1:40659 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40659): Connection reset by peer (error 104)
2026-08-10T06:24:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1496 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:29 dist-test-slave-q9h9 krb5kdc[3179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:29 dist-test-slave-q9h9 krb5kdc[3179](info): set up 2 sockets
Aug 10 06:24:29 dist-test-slave-q9h9 krb5kdc[3179](info): commencing operation
krb5kdc: starting...
W20260810 06:24:30.644366 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:30 dist-test-slave-q9h9 krb5kdc[3179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:24:30.754326 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:30.858469 3203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:30.858685 3203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:30.858736 3203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:30.862603 3203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:30.862650 3203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:30.862668 3203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:30.862694 3203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:30.862722 3203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:30.867682 3203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:30.868551 3203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:30.869462 3203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:30.877329 3214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:30.877365 3216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:30.877341 3213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:30.877483 3203 server_base.cc:1034] running on GCE node
I20260810 06:24:30.878180 3203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:30.879148 3203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:30.880313 3203 hybrid_clock.cc:648] HybridClock initialized: now 1786343070880294 us; error 34 us; skew 500 ppm
Aug 10 06:24:30 dist-test-slave-q9h9 krb5kdc[3179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:30.887575 3203 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:24:30.888445 3203 webserver.cc:460] Webserver started at http://127.0.0.1:34477/ using document root <none> and password file <none>
I20260810 06:24:30.888964 3203 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:30.889030 3203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:30.889221 3203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:30.891227 3203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "bf2c2ed7567a4ed5a256655346e52ef1"
format_stamp: "Formatted at 2026-08-10 06:24:30 on dist-test-slave-q9h9"
server_key: "b0b63685e0983bf2f8a9de9d6298e18d"
server_key_iv: "f26679b537958e8b0b37b629f58335cc"
server_key_version: "encryptionkey@0"
I20260810 06:24:30.891752 3203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "bf2c2ed7567a4ed5a256655346e52ef1"
format_stamp: "Formatted at 2026-08-10 06:24:30 on dist-test-slave-q9h9"
server_key: "b0b63685e0983bf2f8a9de9d6298e18d"
server_key_iv: "f26679b537958e8b0b37b629f58335cc"
server_key_version: "encryptionkey@0"
I20260810 06:24:30.895259 3203 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:24:30.897447 3225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:30.898568 3203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:24:30.898676 3203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "bf2c2ed7567a4ed5a256655346e52ef1"
format_stamp: "Formatted at 2026-08-10 06:24:30 on dist-test-slave-q9h9"
server_key: "b0b63685e0983bf2f8a9de9d6298e18d"
server_key_iv: "f26679b537958e8b0b37b629f58335cc"
server_key_version: "encryptionkey@0"
I20260810 06:24:30.898764 3203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:30.911018 3203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:30.913828 3203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:30.913991 3203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:30.920598 3203 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32775
I20260810 06:24:30.920612 3320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32775 every 8 connection(s)
I20260810 06:24:30.921459 3203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:30.924048 3322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:30.929243 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 3203
I20260810 06:24:30.929383 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:30.929538 3322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1: Bootstrap starting.
I20260810 06:24:30.929698 32563 external_mini_cluster.cc:1428] Setting key 9a9c1cafcab211d8d283f4b748b2cba7
I20260810 06:24:30.932019 3322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:30.932780 3322 log.cc:824] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:30.934823 3322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1: No bootstrap required, opened a new log
I20260810 06:24:30.938330 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:30.930959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:60705 (local address 127.0.0.1:32775)
0810 06:24:30.931347 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:30.931361 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:30.931395 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:24:30.931979 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:30.931993 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:30.932349 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:30.932608 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:30.933002 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:30.934149 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:30.935334 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:30.935780 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:30.935892 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:30.936710 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:30.936715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:30.936726 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:30.937075 (+ 349us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:30.937126 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:30.937146 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:30.937260 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:30.937454 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 06:24:30.937573 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":100,"threads_started":1}
I20260810 06:24:30.938689 3322 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf2c2ed7567a4ed5a256655346e52ef1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32775 } }
I20260810 06:24:30.938901 3322 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:30.938938 3322 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2c2ed7567a4ed5a256655346e52ef1, State: Initialized, Role: FOLLOWER
I20260810 06:24:30.939356 3322 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf2c2ed7567a4ed5a256655346e52ef1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32775 } }
I20260810 06:24:30.939512 3322 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:30.939574 3322 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:30.939668 3322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:30.940601 3322 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf2c2ed7567a4ed5a256655346e52ef1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32775 } }
I20260810 06:24:30.940943 3322 leader_election.cc:302] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf2c2ed7567a4ed5a256655346e52ef1; no voters:
I20260810 06:24:30.941108 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:24:30.941267 3322 leader_election.cc:288] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:30.941339 3329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:30.941529 3329 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1 [term 1 LEADER]: Becoming Leader. State: Replica: bf2c2ed7567a4ed5a256655346e52ef1, State: Running, Role: LEADER
I20260810 06:24:30.941886 3329 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf2c2ed7567a4ed5a256655346e52ef1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32775 } }
I20260810 06:24:30.942514 3322 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:30.943444 3331 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf2c2ed7567a4ed5a256655346e52ef1. Latest consensus state: current_term: 1 leader_uuid: "bf2c2ed7567a4ed5a256655346e52ef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf2c2ed7567a4ed5a256655346e52ef1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32775 } } }
I20260810 06:24:30.943580 3331 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:30.943905 3330 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf2c2ed7567a4ed5a256655346e52ef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf2c2ed7567a4ed5a256655346e52ef1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32775 } } }
I20260810 06:24:30.943971 3345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:30.943986 3330 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:30.946992 3345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:30.949075 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:30.943180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:33747 (local address 127.0.0.1:32775)
0810 06:24:30.943277 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:30.943281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:30.944892 (+ 1611us) server_negotiation.cc:407] Connection header received
0810 06:24:30.944989 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:30.944993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:30.945036 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:30.945120 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:30.945402 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:30.946150 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:30.947328 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:30.947633 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:30.947730 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:30.948606 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:30.948610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:30.948612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:30.948624 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:30.948650 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:30.948661 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:30.948744 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:30.948920 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 06:24:30.948953 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 06:24:30.952243 3345 catalog_manager.cc:1347] Generated new cluster ID: bb3c7be1fec04d8caf4bd17255dbd7df
I20260810 06:24:30.952307 3345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:30.961973 3345 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:30.962815 3345 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:30.974951 3345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf2c2ed7567a4ed5a256655346e52ef1: Generated new TSK 0
I20260810 06:24:30.975540 3345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:30.979586 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 3203
W20260810 06:24:31.001348 3176 connection.cc:535] client connection to 127.0.0.1:32775 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32775): Connection reset by peer (error 104)
2026-08-10T06:24:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1512 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:31 dist-test-slave-q9h9 krb5kdc[3361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:31 dist-test-slave-q9h9 krb5kdc[3361](info): set up 2 sockets
Aug 10 06:24:31 dist-test-slave-q9h9 krb5kdc[3361](info): commencing operation
krb5kdc: starting...
W20260810 06:24:32.145035 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:32 dist-test-slave-q9h9 krb5kdc[3361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:24:32.254030 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38833
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:32.359295 3380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:32.359532 3380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:32.359570 3380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:32.363438 3380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:32.363487 3380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:32.363504 3380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:32.363520 3380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:32.363547 3380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:32.368525 3380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38833
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:32.369446 3380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:32.370414 3380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:32.378208 3386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:32.378225 3388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:32.378247 3392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:32.378496 3380 server_base.cc:1034] running on GCE node
I20260810 06:24:32.379025 3380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:32.380033 3380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:32.381188 3380 hybrid_clock.cc:648] HybridClock initialized: now 1786343072381165 us; error 39 us; skew 500 ppm
Aug 10 06:24:32 dist-test-slave-q9h9 krb5kdc[3361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:32.383744 3380 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:24:32.384555 3380 webserver.cc:460] Webserver started at http://127.0.0.1:37235/ using document root <none> and password file <none>
I20260810 06:24:32.385061 3380 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:32.385124 3380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:32.385310 3380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:32.387251 3380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "60fd21c9caf444f59ad4fb45c6e70e9f"
format_stamp: "Formatted at 2026-08-10 06:24:32 on dist-test-slave-q9h9"
server_key: "40f38db6fa71c15da6b4a47044ce8ba9"
server_key_iv: "0912f15533c1b91b62fef98953a7bc85"
server_key_version: "encryptionkey@0"
I20260810 06:24:32.387857 3380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "60fd21c9caf444f59ad4fb45c6e70e9f"
format_stamp: "Formatted at 2026-08-10 06:24:32 on dist-test-slave-q9h9"
server_key: "40f38db6fa71c15da6b4a47044ce8ba9"
server_key_iv: "0912f15533c1b91b62fef98953a7bc85"
server_key_version: "encryptionkey@0"
I20260810 06:24:32.391350 3380 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:24:32.393541 3406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:32.394690 3380 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:32.394799 3380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "60fd21c9caf444f59ad4fb45c6e70e9f"
format_stamp: "Formatted at 2026-08-10 06:24:32 on dist-test-slave-q9h9"
server_key: "40f38db6fa71c15da6b4a47044ce8ba9"
server_key_iv: "0912f15533c1b91b62fef98953a7bc85"
server_key_version: "encryptionkey@0"
I20260810 06:24:32.394898 3380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:32.399151 3380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:32.401636 3380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:32.401799 3380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:32.408360 3380 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38833
I20260810 06:24:32.408430 3499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38833 every 8 connection(s)
I20260810 06:24:32.409216 3380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:32.411847 3500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:32.417086 3500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f: Bootstrap starting.
I20260810 06:24:32.419286 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 3380
I20260810 06:24:32.419415 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:32.419492 3500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:32.419687 32563 external_mini_cluster.cc:1428] Setting key 6ad9a79cd05beb778c9e8e5a6ee4a183
I20260810 06:24:32.420231 3500 log.cc:824] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:32.422133 3500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f: No bootstrap required, opened a new log
I20260810 06:24:32.424234 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:32.420704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:45987 (local address 127.0.0.1:38833)
0810 06:24:32.421125 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:32.421140 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:32.421176 (+ 36us) server_negotiation.cc:407] Connection header received
0810 06:24:32.421783 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:32.421800 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:32.422088 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:32.422377 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:32.423120 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:32.423125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:32.423135 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:32.423146 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:32.423197 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:32.423214 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:32.423291 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:32.423413 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 06:24:32.423550 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":107,"threads_started":1}
I20260810 06:24:32.426008 3500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60fd21c9caf444f59ad4fb45c6e70e9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38833 } }
I20260810 06:24:32.426311 3500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:32.426371 3500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60fd21c9caf444f59ad4fb45c6e70e9f, State: Initialized, Role: FOLLOWER
I20260810 06:24:32.426779 3500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60fd21c9caf444f59ad4fb45c6e70e9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38833 } }
I20260810 06:24:32.426967 3500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:32.427004 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:24:32.427047 3500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:32.427148 3500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:32.428169 3500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60fd21c9caf444f59ad4fb45c6e70e9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38833 } }
I20260810 06:24:32.428534 3500 leader_election.cc:302] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60fd21c9caf444f59ad4fb45c6e70e9f; no voters:
I20260810 06:24:32.428843 3500 leader_election.cc:288] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:32.428916 3516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:32.429128 3516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f [term 1 LEADER]: Becoming Leader. State: Replica: 60fd21c9caf444f59ad4fb45c6e70e9f, State: Running, Role: LEADER
I20260810 06:24:32.429453 3516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60fd21c9caf444f59ad4fb45c6e70e9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38833 } }
I20260810 06:24:32.430137 3500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:32.431118 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:32.429491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:45131 (local address 127.0.0.1:38833)
0810 06:24:32.429588 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:32.429592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:32.429605 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:24:32.429743 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:32.429746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:32.429778 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:32.429825 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:32.430639 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:32.430642 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:32.430653 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:32.430665 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:32.430692 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:32.430706 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:32.430792 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:32.430926 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 06:24:32.430969 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 06:24:32.431015 3518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60fd21c9caf444f59ad4fb45c6e70e9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60fd21c9caf444f59ad4fb45c6e70e9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38833 } } }
I20260810 06:24:32.431499 3518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:32.431710 3519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60fd21c9caf444f59ad4fb45c6e70e9f. Latest consensus state: current_term: 1 leader_uuid: "60fd21c9caf444f59ad4fb45c6e70e9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60fd21c9caf444f59ad4fb45c6e70e9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38833 } } }
I20260810 06:24:32.431855 3519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:32.431908 3535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:32.434711 3535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:32.439955 3535 catalog_manager.cc:1347] Generated new cluster ID: 7881b7f4a4ed47a78b7d897ea5fa296a
I20260810 06:24:32.440004 3535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:32.457960 3535 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:32.458273 3535 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:32.463215 3535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60fd21c9caf444f59ad4fb45c6e70e9f: Generated new TSK 0
I20260810 06:24:32.463814 3535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:32.476029 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 3380
2026-08-10T06:24:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1484 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:32 dist-test-slave-q9h9 krb5kdc[3555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:32 dist-test-slave-q9h9 krb5kdc[3555](info): set up 2 sockets
Aug 10 06:24:32 dist-test-slave-q9h9 krb5kdc[3555](info): commencing operation
krb5kdc: starting...
W20260810 06:24:33.631770 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:33 dist-test-slave-q9h9 krb5kdc[3555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:24:33.744714 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:38153
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:37947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:38153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:33.852082 3573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:33.852319 3573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:33.852358 3573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:33.856212 3573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:33.856284 3573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:33.856304 3573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:33.856319 3573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:33.856333 3573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:33.861294 3573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:37947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:38153
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:38153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:33.862221 3573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:33.863147 3573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:33.870865 3581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:33.870846 3586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:33.870847 3588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:33.871227 3573 server_base.cc:1034] running on GCE node
I20260810 06:24:33.871732 3573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:33.872761 3573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:33.873927 3573 hybrid_clock.cc:648] HybridClock initialized: now 1786343073873905 us; error 31 us; skew 500 ppm
Aug 10 06:24:33 dist-test-slave-q9h9 krb5kdc[3555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343073, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:33.876520 3573 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:24:33.877314 3573 webserver.cc:460] Webserver started at http://127.31.204.254:38769/ using document root <none> and password file <none>
I20260810 06:24:33.877790 3573 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:33.877847 3573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:33.878003 3573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:33.879990 3573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "601eef04fdc84dbb90caf3b76986a234"
format_stamp: "Formatted at 2026-08-10 06:24:33 on dist-test-slave-q9h9"
server_key: "7efeea3990f0f7de474d0f40515de9b0"
server_key_iv: "bf81a5f9faf68e8bbcc7624149dc3ec9"
server_key_version: "encryptionkey@0"
I20260810 06:24:33.880494 3573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "601eef04fdc84dbb90caf3b76986a234"
format_stamp: "Formatted at 2026-08-10 06:24:33 on dist-test-slave-q9h9"
server_key: "7efeea3990f0f7de474d0f40515de9b0"
server_key_iv: "bf81a5f9faf68e8bbcc7624149dc3ec9"
server_key_version: "encryptionkey@0"
I20260810 06:24:33.883920 3573 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:24:33.886143 3598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:33.887259 3573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:24:33.887362 3573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "601eef04fdc84dbb90caf3b76986a234"
format_stamp: "Formatted at 2026-08-10 06:24:33 on dist-test-slave-q9h9"
server_key: "7efeea3990f0f7de474d0f40515de9b0"
server_key_iv: "bf81a5f9faf68e8bbcc7624149dc3ec9"
server_key_version: "encryptionkey@0"
I20260810 06:24:33.887455 3573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:33.905735 3573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:33.908674 3573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:33.908838 3573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:33.915344 3573 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:38153
I20260810 06:24:33.915357 3686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:38153 every 8 connection(s)
I20260810 06:24:33.916241 3573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:33.918875 3687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:33.920106 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 3573
I20260810 06:24:33.920239 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:33.920481 32563 external_mini_cluster.cc:1428] Setting key 54d4c013badaddf46d67256a7b77c39a
I20260810 06:24:33.925107 3687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234: Bootstrap starting.
I20260810 06:24:33.927600 3687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:33.928315 3687 log.cc:824] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234: Log is configured to *not* fsync() on all Append() calls
Aug 10 06:24:33 dist-test-slave-q9h9 krb5kdc[3555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:24:33.930209 3687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234: No bootstrap required, opened a new log
I20260810 06:24:33.934592 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:33.921778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34539 (local address 127.31.204.254:38153)
0810 06:24:33.922196 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:33.922213 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:33.922255 (+ 42us) server_negotiation.cc:407] Connection header received
0810 06:24:33.923052 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:33.923073 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:33.923483 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:33.923840 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:33.924488 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:33.925862 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:33.926732 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:33.927376 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:33.927528 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:33.929803 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:33.929809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:33.929824 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:33.929853 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:33.931667 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:33.932134 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:33.932140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:33.932146 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:33.932290 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:33.932569 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:33.932572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:33.932574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:33.933053 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:33.933264 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:33.933428 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 06:24:33.933639 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":97,"threads_started":1}
I20260810 06:24:33.934556 3687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601eef04fdc84dbb90caf3b76986a234" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38153 } }
I20260810 06:24:33.934846 3687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:33.934908 3687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601eef04fdc84dbb90caf3b76986a234, State: Initialized, Role: FOLLOWER
I20260810 06:24:33.935334 3687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601eef04fdc84dbb90caf3b76986a234" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38153 } }
I20260810 06:24:33.935452 3687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:33.935534 3687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:33.935634 3687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:33.936448 3687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601eef04fdc84dbb90caf3b76986a234" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38153 } }
I20260810 06:24:33.936758 3687 leader_election.cc:302] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 601eef04fdc84dbb90caf3b76986a234; no voters:
I20260810 06:24:33.937047 3687 leader_election.cc:288] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:33.937119 3694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:33.937316 3694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234 [term 1 LEADER]: Becoming Leader. State: Replica: 601eef04fdc84dbb90caf3b76986a234, State: Running, Role: LEADER
I20260810 06:24:33.937453 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:24:33.937649 3694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601eef04fdc84dbb90caf3b76986a234" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38153 } }
I20260810 06:24:33.938273 3687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:33.939105 3695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "601eef04fdc84dbb90caf3b76986a234" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601eef04fdc84dbb90caf3b76986a234" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38153 } } }
I20260810 06:24:33.939285 3695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:33.939586 3695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 601eef04fdc84dbb90caf3b76986a234. Latest consensus state: current_term: 1 leader_uuid: "601eef04fdc84dbb90caf3b76986a234" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601eef04fdc84dbb90caf3b76986a234" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38153 } } }
I20260810 06:24:33.939733 3695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:33.939774 3719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:33.942939 3719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:33.948649 3719 catalog_manager.cc:1347] Generated new cluster ID: a12ba0632685438bac75b37c9c8fdd69
I20260810 06:24:33.948702 3719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:33.948954 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:33.940598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59627 (local address 127.31.204.254:38153)
0810 06:24:33.940691 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:33.940695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:33.941042 (+ 347us) server_negotiation.cc:407] Connection header received
0810 06:24:33.941147 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:33.941150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:33.941189 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:33.941250 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:33.942063 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:33.942559 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:33.943770 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:33.944013 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:33.944097 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:33.945451 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:33.945455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:33.945457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:33.945482 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:33.947069 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:33.947545 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:33.947549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:33.947551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:33.947625 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:33.947925 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:33.947928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:33.947929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:33.948084 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:33.948217 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:33.948729 (+ 512us) server_negotiation.cc:299] Negotiation successful
0810 06:24:33.948820 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 06:24:33.966183 3719 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:33.967015 3719 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:33.973688 3719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 601eef04fdc84dbb90caf3b76986a234: Generated new TSK 0
I20260810 06:24:33.974350 3719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:33.985066 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 3573
W20260810 06:24:33.999277 3552 connection.cc:535] client connection to 127.31.204.254:38153 recv error: Network error: failed to read from TLS socket (remote: 127.31.204.254:38153): Connection reset by peer (error 104)
2026-08-10T06:24:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1512 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:34 dist-test-slave-q9h9 krb5kdc[3741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:34 dist-test-slave-q9h9 krb5kdc[3741](info): set up 2 sockets
Aug 10 06:24:34 dist-test-slave-q9h9 krb5kdc[3741](info): commencing operation
krb5kdc: starting...
W20260810 06:24:35.158365 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:35 dist-test-slave-q9h9 krb5kdc[3741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:24:35.270634 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:45711
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:45711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:35.378373 3764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:35.378623 3764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:35.378661 3764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:35.382581 3764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:35.382632 3764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:35.382649 3764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:35.382664 3764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:35.382683 3764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:35.387676 3764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:45711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:45711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:35.388583 3764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:35.389530 3764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:35.398063 3772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:35.398111 3764 server_base.cc:1034] running on GCE node
W20260810 06:24:35.398084 3773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:35.398064 3775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:35.398782 3764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:35.399827 3764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:35.400983 3764 hybrid_clock.cc:648] HybridClock initialized: now 1786343075400970 us; error 32 us; skew 500 ppm
Aug 10 06:24:35 dist-test-slave-q9h9 krb5kdc[3741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343075, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:35.403625 3764 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:24:35.404450 3764 webserver.cc:460] Webserver started at http://127.31.204.254:39783/ using document root <none> and password file <none>
I20260810 06:24:35.404955 3764 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:35.405020 3764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:35.405225 3764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:35.407296 3764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "5f2c35578ecb407a993615a8a1438a84"
format_stamp: "Formatted at 2026-08-10 06:24:35 on dist-test-slave-q9h9"
server_key: "fc8aeb5e2ee3608305b57241bd611598"
server_key_iv: "060af55d7b6efea88edb35414557ba41"
server_key_version: "encryptionkey@0"
I20260810 06:24:35.407796 3764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "5f2c35578ecb407a993615a8a1438a84"
format_stamp: "Formatted at 2026-08-10 06:24:35 on dist-test-slave-q9h9"
server_key: "fc8aeb5e2ee3608305b57241bd611598"
server_key_iv: "060af55d7b6efea88edb35414557ba41"
server_key_version: "encryptionkey@0"
I20260810 06:24:35.411406 3764 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:24:35.413905 3786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:35.415095 3764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:24:35.415213 3764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "5f2c35578ecb407a993615a8a1438a84"
format_stamp: "Formatted at 2026-08-10 06:24:35 on dist-test-slave-q9h9"
server_key: "fc8aeb5e2ee3608305b57241bd611598"
server_key_iv: "060af55d7b6efea88edb35414557ba41"
server_key_version: "encryptionkey@0"
I20260810 06:24:35.415321 3764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:35.437006 3764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:35.439823 3764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:35.439971 3764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:35.446473 3764 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:45711
I20260810 06:24:35.446502 3879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:45711 every 8 connection(s)
I20260810 06:24:35.447382 3764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:35.450065 3881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:35.455495 3881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84: Bootstrap starting.
I20260810 06:24:35.455884 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 3764
I20260810 06:24:35.456001 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:35.456250 32563 external_mini_cluster.cc:1428] Setting key d6a0c17404c94aa92f9f586b974b3fb2
I20260810 06:24:35.458053 3881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:35.458767 3881 log.cc:824] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:35.460546 3881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84: No bootstrap required, opened a new log
I20260810 06:24:35.464615 3881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2c35578ecb407a993615a8a1438a84" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45711 } }
I20260810 06:24:35.464829 3881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:35.464886 3881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f2c35578ecb407a993615a8a1438a84, State: Initialized, Role: FOLLOWER
I20260810 06:24:35.464965 3885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:35.457487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48083 (local address 127.31.204.254:45711)
0810 06:24:35.457928 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:35.457942 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:35.457975 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:24:35.458637 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:35.458655 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:35.458998 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:35.459295 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:35.459677 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:35.460610 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:35.461855 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:35.462339 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:35.462456 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:35.463342 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:35.463346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:35.463358 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:35.463708 (+ 350us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:35.463759 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:35.463780 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:35.463876 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:35.464074 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 06:24:35.464203 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":137,"threads_started":1}
I20260810 06:24:35.465397 3881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2c35578ecb407a993615a8a1438a84" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45711 } }
I20260810 06:24:35.465557 3881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:35.465621 3881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:35.465711 3881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:35.466609 3881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2c35578ecb407a993615a8a1438a84" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45711 } }
I20260810 06:24:35.466952 3881 leader_election.cc:302] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f2c35578ecb407a993615a8a1438a84; no voters:
I20260810 06:24:35.467252 3881 leader_election.cc:288] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:35.467324 3888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:35.467536 3888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84 [term 1 LEADER]: Becoming Leader. State: Replica: 5f2c35578ecb407a993615a8a1438a84, State: Running, Role: LEADER
I20260810 06:24:35.467821 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:24:35.468000 3888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2c35578ecb407a993615a8a1438a84" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45711 } }
I20260810 06:24:35.468508 3881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:35.469919 3890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f2c35578ecb407a993615a8a1438a84. Latest consensus state: current_term: 1 leader_uuid: "5f2c35578ecb407a993615a8a1438a84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2c35578ecb407a993615a8a1438a84" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45711 } } }
I20260810 06:24:35.470062 3890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:35.470384 3889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f2c35578ecb407a993615a8a1438a84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2c35578ecb407a993615a8a1438a84" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45711 } } }
I20260810 06:24:35.470480 3889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:35.470815 3913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:35.473903 3913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:35.474682 3885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:35.470947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34311 (local address 127.31.204.254:45711)
0810 06:24:35.471057 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:35.471061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:35.471222 (+ 161us) server_negotiation.cc:407] Connection header received
0810 06:24:35.471348 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:35.471351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:35.471410 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:35.471491 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:35.471804 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:35.472313 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:35.473151 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:35.473379 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:35.473464 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:35.474266 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:35.474270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:35.474272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:35.474283 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:35.474306 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:35.474316 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:35.474388 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:35.474525 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 06:24:35.474557 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:24:35.479727 3913 catalog_manager.cc:1347] Generated new cluster ID: 0691c9c2f3104d469528398e953ef718
I20260810 06:24:35.479781 3913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:35.490273 3913 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:35.491415 3913 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:35.499385 3913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f2c35578ecb407a993615a8a1438a84: Generated new TSK 0
I20260810 06:24:35.500084 3913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:35.507884 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 3764
W20260810 06:24:35.521553 3738 connection.cc:535] client connection to 127.31.204.254:45711 recv error: Network error: failed to read from TLS socket (remote: 127.31.204.254:45711): Connection reset by peer (error 104)
2026-08-10T06:24:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1522 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:35 dist-test-slave-q9h9 krb5kdc[3932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:35 dist-test-slave-q9h9 krb5kdc[3932](info): set up 2 sockets
Aug 10 06:24:35 dist-test-slave-q9h9 krb5kdc[3932](info): commencing operation
krb5kdc: starting...
W20260810 06:24:36.661603 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:36 dist-test-slave-q9h9 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:24:36.770831 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:45345
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:34133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:45345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:36.875663 3951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:36.875888 3951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:36.875941 3951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:36.879770 3951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:36.879819 3951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:36.879845 3951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:36.879866 3951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:36.879890 3951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:36.884866 3951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:34133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:45345
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:45345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:36.885748 3951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:36.886662 3951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:36.894266 3961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:36.894407 3966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:36.894275 3960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:36.894316 3951 server_base.cc:1034] running on GCE node
I20260810 06:24:36.894994 3951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:36.895910 3951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:36.897050 3951 hybrid_clock.cc:648] HybridClock initialized: now 1786343076897033 us; error 30 us; skew 500 ppm
Aug 10 06:24:36 dist-test-slave-q9h9 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343076, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:36.903895 3951 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:24:36.904742 3951 webserver.cc:460] Webserver started at http://127.31.204.254:39501/ using document root <none> and password file <none>
I20260810 06:24:36.905275 3951 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:36.905340 3951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:36.905529 3951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:36.907449 3951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "6f825c64198b4d47a9732ac97058e8ea"
format_stamp: "Formatted at 2026-08-10 06:24:36 on dist-test-slave-q9h9"
server_key: "6097adbb2b8cfb6a226d89c2a83920bd"
server_key_iv: "06dc8882309212c1b7015e32187f1cc8"
server_key_version: "encryptionkey@0"
I20260810 06:24:36.907954 3951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "6f825c64198b4d47a9732ac97058e8ea"
format_stamp: "Formatted at 2026-08-10 06:24:36 on dist-test-slave-q9h9"
server_key: "6097adbb2b8cfb6a226d89c2a83920bd"
server_key_iv: "06dc8882309212c1b7015e32187f1cc8"
server_key_version: "encryptionkey@0"
I20260810 06:24:36.911283 3951 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 06:24:36.913352 3974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:36.914443 3951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:24:36.914565 3951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "6f825c64198b4d47a9732ac97058e8ea"
format_stamp: "Formatted at 2026-08-10 06:24:36 on dist-test-slave-q9h9"
server_key: "6097adbb2b8cfb6a226d89c2a83920bd"
server_key_iv: "06dc8882309212c1b7015e32187f1cc8"
server_key_version: "encryptionkey@0"
I20260810 06:24:36.914649 3951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:36.918779 3951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:36.921149 3951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:36.921303 3951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:36.927511 3951 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:45345
I20260810 06:24:36.927560 4055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:45345 every 8 connection(s)
I20260810 06:24:36.928306 3951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:36.930800 4057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:36.935565 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 3951
I20260810 06:24:36.935694 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:36.935890 4057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea: Bootstrap starting.
I20260810 06:24:36.935937 32563 external_mini_cluster.cc:1428] Setting key 4abd879101a6d1400847a3e882130a97
I20260810 06:24:36.938419 4057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:36.939268 4057 log.cc:824] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:36.940714 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:36.937058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43987 (local address 127.31.204.254:45345)
0810 06:24:36.937488 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:36.937505 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:36.937544 (+ 39us) server_negotiation.cc:407] Connection header received
0810 06:24:36.938329 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:36.938351 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:36.938708 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:36.939044 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:36.939616 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:36.939622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:36.939636 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:36.939648 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:36.939709 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:36.939729 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:36.939826 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:36.939952 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 06:24:36.940093 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":108,"threads_started":1}
I20260810 06:24:36.941114 4057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea: No bootstrap required, opened a new log
I20260810 06:24:36.943323 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:24:36.944870 4057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f825c64198b4d47a9732ac97058e8ea" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45345 } }
I20260810 06:24:36.945071 4057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:36.945166 4057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f825c64198b4d47a9732ac97058e8ea, State: Initialized, Role: FOLLOWER
I20260810 06:24:36.945569 4057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f825c64198b4d47a9732ac97058e8ea" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45345 } }
I20260810 06:24:36.945703 4057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:36.945767 4057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:36.945863 4057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:36.946651 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:36.945161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38961 (local address 127.31.204.254:45345)
0810 06:24:36.945275 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:36.945279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:36.945476 (+ 197us) server_negotiation.cc:407] Connection header received
0810 06:24:36.945576 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:36.945579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:36.945608 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:36.945666 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:36.946330 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:36.946333 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:36.946335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:36.946347 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:36.946370 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:36.946380 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:36.946448 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:36.946521 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 06:24:36.946554 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:24:36.946740 4057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f825c64198b4d47a9732ac97058e8ea" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45345 } }
I20260810 06:24:36.947070 4057 leader_election.cc:302] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f825c64198b4d47a9732ac97058e8ea; no voters:
I20260810 06:24:36.947368 4057 leader_election.cc:288] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:36.947472 4070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:36.947736 4070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea [term 1 LEADER]: Becoming Leader. State: Replica: 6f825c64198b4d47a9732ac97058e8ea, State: Running, Role: LEADER
I20260810 06:24:36.948065 4070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f825c64198b4d47a9732ac97058e8ea" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45345 } }
I20260810 06:24:36.948467 4057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:36.949489 4073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f825c64198b4d47a9732ac97058e8ea. Latest consensus state: current_term: 1 leader_uuid: "6f825c64198b4d47a9732ac97058e8ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f825c64198b4d47a9732ac97058e8ea" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45345 } } }
I20260810 06:24:36.949606 4073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:36.949956 4072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f825c64198b4d47a9732ac97058e8ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f825c64198b4d47a9732ac97058e8ea" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 45345 } } }
I20260810 06:24:36.949994 4083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:36.950145 4072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:36.952600 4083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:36.957413 4083 catalog_manager.cc:1347] Generated new cluster ID: 63467dbeb22d4cb0b1c351087bbb8259
I20260810 06:24:36.957465 4083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:36.969736 4083 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:36.969981 4083 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:36.976624 4083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f825c64198b4d47a9732ac97058e8ea: Generated new TSK 0
I20260810 06:24:36.977236 4083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:36.988467 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 3951
2026-08-10T06:24:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1483 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:37 dist-test-slave-q9h9 krb5kdc[4100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:37 dist-test-slave-q9h9 krb5kdc[4100](info): set up 2 sockets
Aug 10 06:24:37 dist-test-slave-q9h9 krb5kdc[4100](info): commencing operation
krb5kdc: starting...
W20260810 06:24:38.147141 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:38 dist-test-slave-q9h9 krb5kdc[4100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:24:38.264302 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:46077
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:38263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:46077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:38.371161 4123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:38.371385 4123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:38.371424 4123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:38.375216 4123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:38.375264 4123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:38.375285 4123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:38.375305 4123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:38.375319 4123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:38.380139 4123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:38263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:46077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:46077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:38.380959 4123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:38.381821 4123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:38.389439 4135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:38.389446 4133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:38.389587 4138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:38.389710 4123 server_base.cc:1034] running on GCE node
I20260810 06:24:38.390215 4123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:38.391170 4123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:38.392323 4123 hybrid_clock.cc:648] HybridClock initialized: now 1786343078392297 us; error 40 us; skew 500 ppm
Aug 10 06:24:38 dist-test-slave-q9h9 krb5kdc[4100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343078, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:38.394835 4123 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:24:38.395614 4123 webserver.cc:460] Webserver started at http://127.31.204.254:41903/ using document root <none> and password file <none>
I20260810 06:24:38.396085 4123 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:38.396147 4123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:38.396322 4123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:38.398183 4123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "9a3de479ae324f9b94ca2c7d681424d1"
format_stamp: "Formatted at 2026-08-10 06:24:38 on dist-test-slave-q9h9"
server_key: "168bf185029b456d89d9af9eced6987f"
server_key_iv: "d769500503e31d7be6eaead2c5031a61"
server_key_version: "encryptionkey@0"
I20260810 06:24:38.398693 4123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "9a3de479ae324f9b94ca2c7d681424d1"
format_stamp: "Formatted at 2026-08-10 06:24:38 on dist-test-slave-q9h9"
server_key: "168bf185029b456d89d9af9eced6987f"
server_key_iv: "d769500503e31d7be6eaead2c5031a61"
server_key_version: "encryptionkey@0"
I20260810 06:24:38.401949 4123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260810 06:24:38.404048 4147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:38.405119 4123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:38.405226 4123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "9a3de479ae324f9b94ca2c7d681424d1"
format_stamp: "Formatted at 2026-08-10 06:24:38 on dist-test-slave-q9h9"
server_key: "168bf185029b456d89d9af9eced6987f"
server_key_iv: "d769500503e31d7be6eaead2c5031a61"
server_key_version: "encryptionkey@0"
I20260810 06:24:38.405316 4123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:38.419693 4123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:38.422335 4123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:38.422489 4123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:38.428656 4123 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:46077
I20260810 06:24:38.428678 4229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:46077 every 8 connection(s)
I20260810 06:24:38.429468 4123 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:38.429718 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 4123
I20260810 06:24:38.429853 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:38.430168 32563 external_mini_cluster.cc:1428] Setting key 3ca1dbaf28b16f47a3f385b4e4fcb255
I20260810 06:24:38.432513 4232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:38.438438 4232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1: Bootstrap starting.
Aug 10 06:24:38 dist-test-slave-q9h9 krb5kdc[4100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:24:38.440560 4232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:38.441332 4232 log.cc:824] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:38.443082 4232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1: No bootstrap required, opened a new log
I20260810 06:24:38.443444 4239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:38.431393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34179 (local address 127.31.204.254:46077)
0810 06:24:38.431775 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:38.431793 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:38.431832 (+ 39us) server_negotiation.cc:407] Connection header received
0810 06:24:38.432600 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:38.432621 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:38.433004 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:38.433343 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:38.433960 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:38.435438 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:38.436237 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:38.436887 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:38.437041 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:38.439345 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:38.439350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:38.439361 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:38.439386 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:38.440845 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:38.441301 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:38.441305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:38.441309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:38.441405 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:38.441683 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:38.441687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:38.441688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:38.442076 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:38.442242 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:38.442517 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 06:24:38.442771 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":108,"threads_started":1}
I20260810 06:24:38.446002 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:24:38.446753 4232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3de479ae324f9b94ca2c7d681424d1" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46077 } }
I20260810 06:24:38.446941 4232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:38.447018 4232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3de479ae324f9b94ca2c7d681424d1, State: Initialized, Role: FOLLOWER
I20260810 06:24:38.447441 4232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3de479ae324f9b94ca2c7d681424d1" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46077 } }
I20260810 06:24:38.447568 4232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:38.447633 4232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:38.447729 4232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:38.448629 4232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3de479ae324f9b94ca2c7d681424d1" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46077 } }
I20260810 06:24:38.448992 4232 leader_election.cc:302] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a3de479ae324f9b94ca2c7d681424d1; no voters:
I20260810 06:24:38.449273 4232 leader_election.cc:288] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:38.449374 4248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:38.449577 4248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3de479ae324f9b94ca2c7d681424d1, State: Running, Role: LEADER
I20260810 06:24:38.449885 4248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3de479ae324f9b94ca2c7d681424d1" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46077 } }
I20260810 06:24:38.450583 4232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:38.451422 4250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a3de479ae324f9b94ca2c7d681424d1. Latest consensus state: current_term: 1 leader_uuid: "9a3de479ae324f9b94ca2c7d681424d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3de479ae324f9b94ca2c7d681424d1" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46077 } } }
I20260810 06:24:38.451551 4250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:38.451992 4262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:38.451535 4249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a3de479ae324f9b94ca2c7d681424d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3de479ae324f9b94ca2c7d681424d1" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 46077 } } }
I20260810 06:24:38.452164 4249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:38.455035 4262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:38.456429 4239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:38.447807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33667 (local address 127.31.204.254:46077)
0810 06:24:38.447907 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:38.447911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:38.448153 (+ 242us) server_negotiation.cc:407] Connection header received
0810 06:24:38.448223 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:38.448226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:38.448265 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:38.448342 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:38.448920 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:38.449697 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:38.450619 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:38.450962 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:38.451050 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:38.452959 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:38.452962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:38.452965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:38.452988 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:38.454486 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:38.454928 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:38.454931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:38.454933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:38.455014 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:38.455335 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:38.455339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:38.455341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:38.455535 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:38.455655 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:38.456176 (+ 521us) server_negotiation.cc:299] Negotiation successful
0810 06:24:38.456293 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 06:24:38.460289 4262 catalog_manager.cc:1347] Generated new cluster ID: 0ca11d9853a64fae8521255ccae2b7a2
I20260810 06:24:38.460350 4262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:38.472607 4262 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:38.473335 4262 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:38.477921 4262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a3de479ae324f9b94ca2c7d681424d1: Generated new TSK 0
I20260810 06:24:38.478456 4262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:38.483644 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 4123
2026-08-10T06:24:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1496 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:38 dist-test-slave-q9h9 krb5kdc[4284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:38 dist-test-slave-q9h9 krb5kdc[4284](info): set up 2 sockets
Aug 10 06:24:38 dist-test-slave-q9h9 krb5kdc[4284](info): commencing operation
krb5kdc: starting...
W20260810 06:24:39.648623 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:39 dist-test-slave-q9h9 krb5kdc[4284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:24:39.764493 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:35335
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:35335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:39.869530 4309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:39.869761 4309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:39.869815 4309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:39.873651 4309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:39.873705 4309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:39.873731 4309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:39.873752 4309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:39.873785 4309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:39.878839 4309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:35335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:35335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:39.879746 4309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:39.880676 4309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:39.888437 4319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:39.888584 4309 server_base.cc:1034] running on GCE node
W20260810 06:24:39.888473 4317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:39.888443 4316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:39.889192 4309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:39.890228 4309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:39.891431 4309 hybrid_clock.cc:648] HybridClock initialized: now 1786343079891409 us; error 36 us; skew 500 ppm
Aug 10 06:24:39 dist-test-slave-q9h9 krb5kdc[4284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343079, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:39.893908 4309 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:24:39.894738 4309 webserver.cc:460] Webserver started at http://127.31.204.254:46439/ using document root <none> and password file <none>
I20260810 06:24:39.895227 4309 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:39.895294 4309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:39.895480 4309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:39.897346 4309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "c277ebd2ce45458d982fa3b16d7ae1d9"
format_stamp: "Formatted at 2026-08-10 06:24:39 on dist-test-slave-q9h9"
server_key: "84319ae0b8dc4b9e689ff4369a40f3d2"
server_key_iv: "27df78d582bbf7b735b98fda6f3686fe"
server_key_version: "encryptionkey@0"
I20260810 06:24:39.897850 4309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "c277ebd2ce45458d982fa3b16d7ae1d9"
format_stamp: "Formatted at 2026-08-10 06:24:39 on dist-test-slave-q9h9"
server_key: "84319ae0b8dc4b9e689ff4369a40f3d2"
server_key_iv: "27df78d582bbf7b735b98fda6f3686fe"
server_key_version: "encryptionkey@0"
I20260810 06:24:39.901330 4309 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:24:39.903492 4330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:39.904606 4309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:24:39.904713 4309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "c277ebd2ce45458d982fa3b16d7ae1d9"
format_stamp: "Formatted at 2026-08-10 06:24:39 on dist-test-slave-q9h9"
server_key: "84319ae0b8dc4b9e689ff4369a40f3d2"
server_key_iv: "27df78d582bbf7b735b98fda6f3686fe"
server_key_version: "encryptionkey@0"
I20260810 06:24:39.904799 4309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:39.921880 4309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:39.924608 4309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:39.924765 4309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:39.931190 4309 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:35335
I20260810 06:24:39.931210 4414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:35335 every 8 connection(s)
I20260810 06:24:39.931993 4309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:39.934638 4415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:39.939663 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 4309
I20260810 06:24:39.939724 4415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9: Bootstrap starting.
I20260810 06:24:39.939801 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:39.940048 32563 external_mini_cluster.cc:1428] Setting key ae1bb0ca92f661b442b5de1cb06ad9f8
I20260810 06:24:39.942062 4415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:39.942765 4415 log.cc:824] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:39.944645 4415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9: No bootstrap required, opened a new log
I20260810 06:24:39.948205 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:39.941220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58449 (local address 127.31.204.254:35335)
0810 06:24:39.941602 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:39.941616 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:39.941649 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:24:39.942299 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:39.942315 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:39.942618 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:39.942877 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:39.943244 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:39.944241 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:39.945429 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:39.945862 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:39.945976 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:39.946744 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:39.946748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:39.946759 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:39.947095 (+ 336us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:39.947141 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:39.947161 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:39.947233 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:39.947385 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 06:24:39.947551 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":100,"threads_started":1}
I20260810 06:24:39.948514 4415 raft_consensus.cc:348] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c277ebd2ce45458d982fa3b16d7ae1d9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35335 } }
I20260810 06:24:39.948711 4415 raft_consensus.cc:374] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:39.948760 4415 raft_consensus.cc:729] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c277ebd2ce45458d982fa3b16d7ae1d9, State: Initialized, Role: FOLLOWER
I20260810 06:24:39.949137 4415 consensus_queue.cc:260] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c277ebd2ce45458d982fa3b16d7ae1d9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35335 } }
I20260810 06:24:39.949268 4415 raft_consensus.cc:388] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:39.949331 4415 raft_consensus.cc:482] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:39.949431 4415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:39.950331 4415 raft_consensus.cc:504] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c277ebd2ce45458d982fa3b16d7ae1d9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35335 } }
I20260810 06:24:39.950654 4415 leader_election.cc:302] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c277ebd2ce45458d982fa3b16d7ae1d9; no voters:
I20260810 06:24:39.950704 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:24:39.950932 4415 leader_election.cc:288] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:39.951028 4426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:39.951220 4426 raft_consensus.cc:686] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9 [term 1 LEADER]: Becoming Leader. State: Replica: c277ebd2ce45458d982fa3b16d7ae1d9, State: Running, Role: LEADER
I20260810 06:24:39.951560 4426 consensus_queue.cc:237] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c277ebd2ce45458d982fa3b16d7ae1d9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35335 } }
I20260810 06:24:39.952018 4415 sys_catalog.cc:564] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:39.952714 4426 sys_catalog.cc:455] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c277ebd2ce45458d982fa3b16d7ae1d9. Latest consensus state: current_term: 1 leader_uuid: "c277ebd2ce45458d982fa3b16d7ae1d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c277ebd2ce45458d982fa3b16d7ae1d9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35335 } } }
I20260810 06:24:39.952895 4426 sys_catalog.cc:458] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:39.953099 4427 sys_catalog.cc:455] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c277ebd2ce45458d982fa3b16d7ae1d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c277ebd2ce45458d982fa3b16d7ae1d9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35335 } } }
I20260810 06:24:39.953200 4427 sys_catalog.cc:458] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:39.953320 4441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:39.956354 4441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:39.957652 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:39.952805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45299 (local address 127.31.204.254:35335)
0810 06:24:39.952900 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:39.952904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:39.953489 (+ 585us) server_negotiation.cc:407] Connection header received
0810 06:24:39.953525 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:39.953527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:39.953563 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:39.953620 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:39.953978 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:39.954698 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:39.955833 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:39.956135 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:39.956235 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:39.957054 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:39.957059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:39.957061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:39.957073 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:39.957102 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:39.957115 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:39.957178 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:39.957468 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 06:24:39.957519 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 06:24:39.961133 4441 catalog_manager.cc:1347] Generated new cluster ID: 3155eebf2d554e97b873d15bfdfe12cb
I20260810 06:24:39.961193 4441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:39.966851 4441 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:39.967640 4441 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:39.974125 4441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c277ebd2ce45458d982fa3b16d7ae1d9: Generated new TSK 0
I20260810 06:24:39.974637 4441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:39.979024 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 4309
2026-08-10T06:24:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1493 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:40 dist-test-slave-q9h9 krb5kdc[4466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:40 dist-test-slave-q9h9 krb5kdc[4466](info): set up 2 sockets
Aug 10 06:24:40 dist-test-slave-q9h9 krb5kdc[4466](info): commencing operation
krb5kdc: starting...
W20260810 06:24:41.134689 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:41 dist-test-slave-q9h9 krb5kdc[4466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:24:41.248992 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:35651
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:35651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:41.353845 4486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:41.354103 4486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:41.354147 4486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:41.357937 4486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:41.357985 4486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:41.358001 4486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:41.358037 4486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:41.358068 4486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:41.362963 4486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:35651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:35651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:41.363780 4486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:41.364636 4486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:41.372129 4500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:41.372355 4486 server_base.cc:1034] running on GCE node
W20260810 06:24:41.372172 4502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:41.372130 4497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:41.373009 4486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:41.374007 4486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:41.375193 4486 hybrid_clock.cc:648] HybridClock initialized: now 1786343081375173 us; error 33 us; skew 500 ppm
Aug 10 06:24:41 dist-test-slave-q9h9 krb5kdc[4466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343081, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:41.377640 4486 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:24:41.378453 4486 webserver.cc:460] Webserver started at http://127.31.204.254:33349/ using document root <none> and password file <none>
I20260810 06:24:41.378930 4486 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:41.378993 4486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:41.379159 4486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:41.380975 4486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "f349af012a4f447e9c9ce2922e2ff4b9"
format_stamp: "Formatted at 2026-08-10 06:24:41 on dist-test-slave-q9h9"
server_key: "94988144ce288123d951d5cc10c5e748"
server_key_iv: "76b08bb10f941f94916cdd5b31348593"
server_key_version: "encryptionkey@0"
I20260810 06:24:41.381470 4486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "f349af012a4f447e9c9ce2922e2ff4b9"
format_stamp: "Formatted at 2026-08-10 06:24:41 on dist-test-slave-q9h9"
server_key: "94988144ce288123d951d5cc10c5e748"
server_key_iv: "76b08bb10f941f94916cdd5b31348593"
server_key_version: "encryptionkey@0"
I20260810 06:24:41.384770 4486 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:24:41.386914 4512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:41.388008 4486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:41.388114 4486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "f349af012a4f447e9c9ce2922e2ff4b9"
format_stamp: "Formatted at 2026-08-10 06:24:41 on dist-test-slave-q9h9"
server_key: "94988144ce288123d951d5cc10c5e748"
server_key_iv: "76b08bb10f941f94916cdd5b31348593"
server_key_version: "encryptionkey@0"
I20260810 06:24:41.388198 4486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:41.392424 4486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:41.395290 4486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:41.395447 4486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:41.401726 4486 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:35651
I20260810 06:24:41.401777 4587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:35651 every 8 connection(s)
I20260810 06:24:41.402591 4486 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:41.403762 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 4486
I20260810 06:24:41.403891 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:41.404166 32563 external_mini_cluster.cc:1428] Setting key beb2ab6ee402ab09f37bffe63aefcd62
I20260810 06:24:41.405576 4588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:41.409298 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:41.405380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54825 (local address 127.31.204.254:35651)
0810 06:24:41.405789 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:41.405807 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:41.405847 (+ 40us) server_negotiation.cc:407] Connection header received
0810 06:24:41.406662 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:41.406682 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:41.407025 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:41.407359 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:41.407965 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:41.407971 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:41.407985 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:41.407996 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:41.408051 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:41.408075 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:41.408218 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:41.408286 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 06:24:41.408423 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":114,"threads_started":1}
I20260810 06:24:41.411772 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:24:41.411864 4588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9: Bootstrap starting.
I20260810 06:24:41.414474 4588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:41.415347 4588 log.cc:824] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:41.415515 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:41.413959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34775 (local address 127.31.204.254:35651)
0810 06:24:41.414086 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:41.414090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:41.414356 (+ 266us) server_negotiation.cc:407] Connection header received
0810 06:24:41.414468 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:41.414470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:41.414497 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:41.414566 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:41.415179 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:41.415181 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:41.415183 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:24:41.415192 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:41.415214 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:24:41.415223 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:41.415288 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:41.415384 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 06:24:41.415410 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:24:41.417176 4588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9: No bootstrap required, opened a new log
I20260810 06:24:41.420708 4588 raft_consensus.cc:348] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f349af012a4f447e9c9ce2922e2ff4b9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35651 } }
I20260810 06:24:41.420907 4588 raft_consensus.cc:374] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:41.420957 4588 raft_consensus.cc:729] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f349af012a4f447e9c9ce2922e2ff4b9, State: Initialized, Role: FOLLOWER
I20260810 06:24:41.421362 4588 consensus_queue.cc:260] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f349af012a4f447e9c9ce2922e2ff4b9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35651 } }
I20260810 06:24:41.421523 4588 raft_consensus.cc:388] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:41.421593 4588 raft_consensus.cc:482] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:41.421689 4588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:41.422572 4588 raft_consensus.cc:504] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f349af012a4f447e9c9ce2922e2ff4b9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35651 } }
I20260810 06:24:41.422850 4588 leader_election.cc:302] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f349af012a4f447e9c9ce2922e2ff4b9; no voters:
I20260810 06:24:41.423084 4588 leader_election.cc:288] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:41.423187 4607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:41.423385 4607 raft_consensus.cc:686] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9 [term 1 LEADER]: Becoming Leader. State: Replica: f349af012a4f447e9c9ce2922e2ff4b9, State: Running, Role: LEADER
I20260810 06:24:41.423715 4607 consensus_queue.cc:237] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f349af012a4f447e9c9ce2922e2ff4b9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35651 } }
I20260810 06:24:41.424161 4588 sys_catalog.cc:564] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:41.425230 4609 sys_catalog.cc:455] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f349af012a4f447e9c9ce2922e2ff4b9. Latest consensus state: current_term: 1 leader_uuid: "f349af012a4f447e9c9ce2922e2ff4b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f349af012a4f447e9c9ce2922e2ff4b9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35651 } } }
I20260810 06:24:41.425357 4609 sys_catalog.cc:458] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:41.425633 4608 sys_catalog.cc:455] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f349af012a4f447e9c9ce2922e2ff4b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f349af012a4f447e9c9ce2922e2ff4b9" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 35651 } } }
I20260810 06:24:41.425731 4608 sys_catalog.cc:458] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:41.425751 4622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:41.428776 4622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:41.433864 4622 catalog_manager.cc:1347] Generated new cluster ID: 95557ae0cdad4de5985b40ae57c97643
I20260810 06:24:41.433913 4622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:41.451604 4622 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:41.451895 4622 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:41.458931 4622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f349af012a4f447e9c9ce2922e2ff4b9: Generated new TSK 0
I20260810 06:24:41.459488 4622 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:24:41.463955 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 4486
2026-08-10T06:24:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1481 ms)
[----------] 18 tests from AuthTokenIssuingTest (26909 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:41 dist-test-slave-q9h9 krb5kdc[4648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:41 dist-test-slave-q9h9 krb5kdc[4648](info): set up 2 sockets
Aug 10 06:24:41 dist-test-slave-q9h9 krb5kdc[4648](info): commencing operation
krb5kdc: starting...
W20260810 06:24:42.620599 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:42 dist-test-slave-q9h9 krb5kdc[4648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:24:42.732156 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:38005
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:36265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:38005,127.31.204.253:39423,127.31.204.252:43047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:42.838503 4675 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 06:24:42.838762 4675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:42.838794 4675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:42.838851 4675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:42.842722 4675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:42.842772 4675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:42.842795 4675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:42.842815 4675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:42.842844 4675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:42.847841 4675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:36265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:38005,127.31.204.253:39423,127.31.204.252:43047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:38005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:42.848793 4675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:42.849736 4675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:42.857611 4684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:42.857684 4675 server_base.cc:1034] running on GCE node
W20260810 06:24:42.857647 4686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:42.857611 4688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:42.858374 4675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:42.859357 4675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:42.860514 4675 hybrid_clock.cc:648] HybridClock initialized: now 1786343082860499 us; error 29 us; skew 500 ppm
Aug 10 06:24:42 dist-test-slave-q9h9 krb5kdc[4648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343082, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:42.866510 4675 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:24:42.867333 4675 webserver.cc:460] Webserver started at http://127.31.204.254:37053/ using document root <none> and password file <none>
I20260810 06:24:42.867833 4675 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:42.867898 4675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:42.868077 4675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:42.870005 4675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "dbfb0a8896894ce88304f04ab545be20"
format_stamp: "Formatted at 2026-08-10 06:24:42 on dist-test-slave-q9h9"
server_key: "eb705506efc2fcfc1d8e17f9d4aa38d6"
server_key_iv: "926251eff0f45f69bdbbf2280fd74ff6"
server_key_version: "encryptionkey@0"
I20260810 06:24:42.870574 4675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "dbfb0a8896894ce88304f04ab545be20"
format_stamp: "Formatted at 2026-08-10 06:24:42 on dist-test-slave-q9h9"
server_key: "eb705506efc2fcfc1d8e17f9d4aa38d6"
server_key_iv: "926251eff0f45f69bdbbf2280fd74ff6"
server_key_version: "encryptionkey@0"
I20260810 06:24:42.873946 4675 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260810 06:24:42.876197 4705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:42.877321 4675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:42.877437 4675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "dbfb0a8896894ce88304f04ab545be20"
format_stamp: "Formatted at 2026-08-10 06:24:42 on dist-test-slave-q9h9"
server_key: "eb705506efc2fcfc1d8e17f9d4aa38d6"
server_key_iv: "926251eff0f45f69bdbbf2280fd74ff6"
server_key_version: "encryptionkey@0"
I20260810 06:24:42.877524 4675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:42.898322 4675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:42.900820 4675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:42.900985 4675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:42.907470 4675 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:38005
I20260810 06:24:42.907727 4795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:38005 every 8 connection(s)
I20260810 06:24:42.908352 4675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:42.910969 4797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:42.914562 4797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:24:42.917375 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 4675
I20260810 06:24:42.917492 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:42.917770 32563 external_mini_cluster.cc:1428] Setting key c15a7f2cc5e8d6d637a43dd3fe8012fc
Aug 10 06:24:42 dist-test-slave-q9h9 krb5kdc[4648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343082, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.204.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.253@KRBTEST.COM" created.
I20260810 06:24:42.929173 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:42.916560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:38005 (local address 127.0.0.1:44950)
0810 06:24:42.917003 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:42.917040 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:42.918076 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:42.918726 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:42.918731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:42.919216 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:42.919542 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:42.919559 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:42.921071 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:42.921074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:42.921771 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:42.921779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:42.922627 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:42.922631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:42.922735 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:42.923231 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:42.923247 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:42.924810 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:42.926839 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:42.926849 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:42.926859 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:42.927067 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:42.927366 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:42.927368 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:42.927369 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:42.927461 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:42.927960 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:42.927965 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:42.928080 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 06:24:42.928210 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 06:24:42.928515 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":77,"thread_start_us":120,"threads_started":1}
I20260810 06:24:42.929174 4803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:42.916560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44950 (local address 127.31.204.254:38005)
0810 06:24:42.916979 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:42.916995 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:42.917130 (+ 135us) server_negotiation.cc:407] Connection header received
0810 06:24:42.918367 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:42.918371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:42.918539 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:42.918681 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:42.919677 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:42.920960 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:42.921885 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:42.922486 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:42.922642 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:42.924965 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:42.924972 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:42.924985 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:42.925012 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:42.926731 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:42.927169 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:42.927174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:42.927175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:42.927295 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:42.927525 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:42.927527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:42.927528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:42.927869 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:42.927968 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:42.928277 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 06:24:42.928433 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"spinlock_wait_cycles":66944,"thread_start_us":159,"threads_started":1}
I20260810 06:24:42.931658 4797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:24:42.932279 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:42.932065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:39423 (local address 127.0.0.1:58620)
0810 06:24:42.932165 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:42.932227 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.253:39423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 06:24:42.932433 4714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.204.253:39423: connect: Connection refused (error 111)
I20260810 06:24:42.933172 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:42.932993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:39423 (local address 127.0.0.1:58630)
0810 06:24:42.933086 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:42.933127 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.253:39423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 06:24:42.933389 4797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.253:39423: Network error: Client connection negotiation failed: client connection to 127.31.204.253:39423: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.253.keytab.
Entry for principal kudu/127.31.204.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.253.keytab.
I20260810 06:24:42.941700 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.253:39423
--webserver_interface=127.31.204.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.253.keytab
--principal=kudu/127.31.204.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:36265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:38005,127.31.204.253:39423,127.31.204.252:43047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:24:42.982583 4797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } attempt: 1
I20260810 06:24:42.983433 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:42.983130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:39423 (local address 127.0.0.1:58632)
0810 06:24:42.983294 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:42.983357 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.253:39423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 06:24:42.984097 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:42.983885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:39423 (local address 127.0.0.1:58648)
0810 06:24:42.983986 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:42.984029 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.253:39423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 06:24:42.984207 4797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.253:39423: Network error: Client connection negotiation failed: client connection to 127.31.204.253:39423: connect: Connection refused (error 111)
I20260810 06:24:43.052357 4797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } attempt: 2
I20260810 06:24:43.053187 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.052895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:39423 (local address 127.0.0.1:58660)
0810 06:24:43.053041 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.053104 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.253:39423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 06:24:43.053880 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.053660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:39423 (local address 127.0.0.1:58676)
0810 06:24:43.053761 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.053805 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.253:39423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 06:24:43.054075 4797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.253:39423: Network error: Client connection negotiation failed: client connection to 127.31.204.253:39423: connect: Connection refused (error 111)
W20260810 06:24:43.063371 4807 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 06:24:43.063606 4807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:43.063632 4807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:43.063673 4807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:43.067562 4807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:43.067607 4807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:43.067625 4807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:43.067641 4807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:43.067653 4807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:43.072695 4807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:36265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:38005,127.31.204.253:39423,127.31.204.252:43047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.253.keytab
--principal=kudu/127.31.204.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.253:39423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.204.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:43.073580 4807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:43.074550 4807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:43.082635 4819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:43.082648 4816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:43.082654 4815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:43.082863 4807 server_base.cc:1034] running on GCE node
I20260810 06:24:43.083474 4807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:43.084519 4807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:43.085677 4807 hybrid_clock.cc:648] HybridClock initialized: now 1786343083085656 us; error 38 us; skew 500 ppm
Aug 10 06:24:43 dist-test-slave-q9h9 krb5kdc[4648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343083, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:43.088260 4807 init.cc:377] Logged in from keytab as kudu/127.31.204.253@KRBTEST.COM (short username kudu)
I20260810 06:24:43.089068 4807 webserver.cc:460] Webserver started at http://127.31.204.253:46479/ using document root <none> and password file <none>
I20260810 06:24:43.089570 4807 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:43.089639 4807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:43.089848 4807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:43.091933 4807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-1/data/instance:
uuid: "86a3314009874036ab6cd2864681b3b9"
format_stamp: "Formatted at 2026-08-10 06:24:43 on dist-test-slave-q9h9"
server_key: "7d0fe560025b286c5ca42f613f8bed31"
server_key_iv: "1d4bc46e081bb7b71d1df3f55e40e1e8"
server_key_version: "encryptionkey@0"
I20260810 06:24:43.092509 4807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-1/wal/instance:
uuid: "86a3314009874036ab6cd2864681b3b9"
format_stamp: "Formatted at 2026-08-10 06:24:43 on dist-test-slave-q9h9"
server_key: "7d0fe560025b286c5ca42f613f8bed31"
server_key_iv: "1d4bc46e081bb7b71d1df3f55e40e1e8"
server_key_version: "encryptionkey@0"
I20260810 06:24:43.096222 4807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:24:43.098519 4827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:43.099644 4807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:24:43.099761 4807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-1/wal
uuid: "86a3314009874036ab6cd2864681b3b9"
format_stamp: "Formatted at 2026-08-10 06:24:43 on dist-test-slave-q9h9"
server_key: "7d0fe560025b286c5ca42f613f8bed31"
server_key_iv: "1d4bc46e081bb7b71d1df3f55e40e1e8"
server_key_version: "encryptionkey@0"
I20260810 06:24:43.099853 4807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:43.112339 4807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:43.114830 4807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:43.114992 4807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:43.121366 4807 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.253:39423
I20260810 06:24:43.121606 4922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.253:39423 every 8 connection(s)
I20260810 06:24:43.122298 4807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-1/data/info.pb
I20260810 06:24:43.124864 4924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:43.127334 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 4807
I20260810 06:24:43.127450 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-1/wal/instance
I20260810 06:24:43.127737 32563 external_mini_cluster.cc:1428] Setting key 5725cf4a28710246768e054b15a1c71b
I20260810 06:24:43.128973 4924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.204.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.252@KRBTEST.COM" created.
Aug 10 06:24:43 dist-test-slave-q9h9 krb5kdc[4648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343083, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.253@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:24:43.144264 4803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.131017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44960 (local address 127.31.204.254:38005)
0810 06:24:43.131141 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.131145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.132115 (+ 970us) server_negotiation.cc:407] Connection header received
0810 06:24:43.133033 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.133037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.133101 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.133172 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.134346 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.134888 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.135979 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.136209 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.136306 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.140041 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:43.140045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:43.140047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:43.140077 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:43.142085 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.142667 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.142670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.142671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.142741 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.143126 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.143127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.143129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.143331 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:43.143406 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:43.144041 (+ 635us) server_negotiation.cc:299] Negotiation successful
0810 06:24:43.144140 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:24:43.145231 4797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } attempt: 3
I20260810 06:24:43.145438 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.131482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:38005 (local address 127.0.0.1:44960)
0810 06:24:43.131970 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.132010 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:43.132813 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:43.133352 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:43.133358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:43.133680 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:43.134206 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:43.134221 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.135009 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.135012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:43.135862 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:43.135871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.136347 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.136352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:43.136511 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.137044 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:43.137061 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:43.139867 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:43.142197 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:43.142203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:43.142219 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:43.142564 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:43.142835 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:43.142838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:43.142841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:43.143043 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:43.143439 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:43.143446 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:43.143787 (+ 341us) client_negotiation.cc:769] Sending connection context
0810 06:24:43.144011 (+ 224us) client_negotiation.cc:240] Negotiation successful
0810 06:24:43.144367 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":145,"threads_started":1}
I20260810 06:24:43.147006 4924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:24:43 dist-test-slave-q9h9 krb5kdc[4648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343082, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for kudu/127.31.204.253@KRBTEST.COM
Aug 10 06:24:43 dist-test-slave-q9h9 krb5kdc[4648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343083, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.253@KRBTEST.COM for kudu/127.31.204.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.252.keytab.
Entry for principal kudu/127.31.204.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.252.keytab.
I20260810 06:24:43.154966 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.252:43047
--webserver_interface=127.31.204.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.252.keytab
--principal=kudu/127.31.204.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:36265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:38005,127.31.204.253:39423,127.31.204.252:43047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:24:43.155493 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.147439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:39423 (local address 127.0.0.1:58696)
0810 06:24:43.147582 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.147595 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:43.147662 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:43.147935 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:43.147938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:43.148076 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:43.148151 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:43.148156 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.148848 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.148851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:43.149612 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:43.149621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.150114 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.150118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:43.150280 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.150667 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:43.150681 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:43.151977 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:43.154058 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:43.154061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:43.154063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:43.154290 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:43.154551 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:43.154554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:43.154555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:43.154628 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:43.155040 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:43.155042 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:43.155124 (+ 82us) client_negotiation.cc:769] Sending connection context
0810 06:24:43.155209 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 06:24:43.155326 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":53}
I20260810 06:24:43.155596 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.147474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.31.204.253:39423)
0810 06:24:43.147765 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.147773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.147783 (+ 10us) server_negotiation.cc:407] Connection header received
0810 06:24:43.147806 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.147808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.147837 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.147909 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.148250 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.148764 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.149707 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.149994 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.150107 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.152123 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:43.152127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:43.152130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:43.152155 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:43.153928 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.154388 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.154391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.154393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.154466 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.154704 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.154707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.154709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.154944 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:43.155058 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:43.155306 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 06:24:43.155426 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":235,"spinlock_wait_cycles":9472,"thread_start_us":98,"threads_started":1}
I20260810 06:24:43.155671 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.145616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:39423 (local address 127.0.0.1:58686)
0810 06:24:43.145812 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.145826 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:43.145902 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:43.146699 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:43.146702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:43.146879 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:43.146995 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:43.147002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.148211 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.148213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:43.148765 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:43.148775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.149523 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.149527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:43.149654 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.150155 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:43.150169 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:43.151477 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:43.154096 (+ 2619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:43.154100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:43.154102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:43.154299 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:43.154634 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:43.154638 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:43.154639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:43.154731 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:43.155193 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:43.155196 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:43.155293 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 06:24:43.155369 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 06:24:43.155510 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260810 06:24:43.156177 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.145910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58686 (local address 127.31.204.253:39423)
0810 06:24:43.146265 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.146276 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.146299 (+ 23us) server_negotiation.cc:407] Connection header received
0810 06:24:43.146341 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.146346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.146543 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.146694 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.147124 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.148086 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.148884 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.149398 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.149497 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.151596 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:43.151604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:43.151614 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:43.151635 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:43.153941 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.154429 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.154432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.154433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.154519 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.154864 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.154867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.154869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.155074 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:43.155847 (+ 773us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:43.155922 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 06:24:43.156026 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":168,"threads_started":1}
I20260810 06:24:43.157295 4924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:24:43.157588 4797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:24:43.157958 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.157686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43047 (local address 127.0.0.1:43738)
0810 06:24:43.157807 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.157889 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 06:24:43.158211 4829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43047: connect: Connection refused (error 111)
I20260810 06:24:43.158316 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.158043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43047 (local address 127.0.0.1:43748)
0810 06:24:43.158212 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.158255 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260810 06:24:43.159001 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.158778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43047 (local address 127.0.0.1:43756)
0810 06:24:43.158909 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.158955 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 06:24:43.159030 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.158809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43047 (local address 127.0.0.1:43768)
0810 06:24:43.158934 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.158975 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260810 06:24:43.159216 4797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.252:43047: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43047: connect: Connection refused (error 111)
W20260810 06:24:43.159171 4924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.252:43047: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43047: connect: Connection refused (error 111)
I20260810 06:24:43.190383 4797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } attempt: 1
I20260810 06:24:43.191232 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.190927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43047 (local address 127.0.0.1:43784)
0810 06:24:43.191075 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.191154 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 06:24:43.191898 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.191692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43047 (local address 127.0.0.1:43792)
0810 06:24:43.191797 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.191842 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 06:24:43.192044 4797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.252:43047: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43047: connect: Connection refused (error 111)
I20260810 06:24:43.208420 4924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } attempt: 1
I20260810 06:24:43.209203 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.208918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43047 (local address 127.0.0.1:43794)
0810 06:24:43.209060 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.209126 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 06:24:43.209887 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.209672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43047 (local address 127.0.0.1:43808)
0810 06:24:43.209779 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.209825 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 06:24:43.209996 4924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.252:43047: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43047: connect: Connection refused (error 111)
W20260810 06:24:43.266569 4938 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 06:24:43.266917 4938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:43.266956 4938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:43.267011 4938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 06:24:43.267189 4797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } attempt: 2
I20260810 06:24:43.268000 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.267693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43047 (local address 127.0.0.1:43822)
0810 06:24:43.267844 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.267912 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 06:24:43.268738 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.268497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43047 (local address 127.0.0.1:43830)
0810 06:24:43.268634 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.268674 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260810 06:24:43.268954 4797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.252:43047: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43047: connect: Connection refused (error 111)
W20260810 06:24:43.272699 4938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:43.272761 4938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:43.272785 4938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:43.272812 4938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:43.272832 4938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:43.278172 4924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } attempt: 2
I20260810 06:24:43.278877 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.278613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43047 (local address 127.0.0.1:43838)
0810 06:24:43.278738 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.278795 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 06:24:43.279594 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.279370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43047 (local address 127.0.0.1:43842)
0810 06:24:43.279475 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.279528 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260810 06:24:43.279755 4924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.252:43047: Network error: Client connection negotiation failed: client connection to 127.31.204.252:43047: connect: Connection refused (error 111)
I20260810 06:24:43.280575 4938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:36265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:38005,127.31.204.253:39423,127.31.204.252:43047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.252.keytab
--principal=kudu/127.31.204.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.252:43047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.204.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:43.281744 4938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:43.282954 4938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:43.291981 4951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:43.291960 4948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:43.292145 4949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:43.292284 4938 server_base.cc:1034] running on GCE node
I20260810 06:24:43.292836 4938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:43.293859 4938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:43.295015 4938 hybrid_clock.cc:648] HybridClock initialized: now 1786343083294987 us; error 38 us; skew 500 ppm
Aug 10 06:24:43 dist-test-slave-q9h9 krb5kdc[4648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343083, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:43.297664 4938 init.cc:377] Logged in from keytab as kudu/127.31.204.252@KRBTEST.COM (short username kudu)
I20260810 06:24:43.298543 4938 webserver.cc:460] Webserver started at http://127.31.204.252:41183/ using document root <none> and password file <none>
I20260810 06:24:43.299041 4938 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:43.299103 4938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:43.299312 4938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:43.301250 4938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-2/data/instance:
uuid: "95bf645f7e364ebdb0c6301d1ef6db90"
format_stamp: "Formatted at 2026-08-10 06:24:43 on dist-test-slave-q9h9"
server_key: "3c23c5e0dab75894c7d79465ab6dd3d0"
server_key_iv: "cfeafe3b99202f02008d9a303b086dfb"
server_key_version: "encryptionkey@0"
I20260810 06:24:43.301772 4938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-2/wal/instance:
uuid: "95bf645f7e364ebdb0c6301d1ef6db90"
format_stamp: "Formatted at 2026-08-10 06:24:43 on dist-test-slave-q9h9"
server_key: "3c23c5e0dab75894c7d79465ab6dd3d0"
server_key_iv: "cfeafe3b99202f02008d9a303b086dfb"
server_key_version: "encryptionkey@0"
I20260810 06:24:43.305238 4938 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:24:43.307475 4961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:43.308578 4938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:24:43.308678 4938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-2/wal
uuid: "95bf645f7e364ebdb0c6301d1ef6db90"
format_stamp: "Formatted at 2026-08-10 06:24:43 on dist-test-slave-q9h9"
server_key: "3c23c5e0dab75894c7d79465ab6dd3d0"
server_key_iv: "cfeafe3b99202f02008d9a303b086dfb"
server_key_version: "encryptionkey@0"
I20260810 06:24:43.308775 4938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:43.327035 4938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:43.329850 4938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:43.330031 4938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:43.336367 4938 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.252:43047
I20260810 06:24:43.336612 5045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.252:43047 every 8 connection(s)
I20260810 06:24:43.337236 4938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-2/data/info.pb
I20260810 06:24:43.339767 5047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:43.342075 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 4938
I20260810 06:24:43.342185 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786342803670820-32563-0/minicluster-data/master-2/wal/instance
I20260810 06:24:43.342464 32563 external_mini_cluster.cc:1428] Setting key 1609efcaf09d72beedfdbe4f8147f9fa
I20260810 06:24:43.343577 5047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:24:43 dist-test-slave-q9h9 krb5kdc[4648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343083, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.252@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
Aug 10 06:24:43 dist-test-slave-q9h9 krb5kdc[4648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.252@KRBTEST.COM
I20260810 06:24:43.354344 4803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.345372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44966 (local address 127.31.204.254:38005)
0810 06:24:43.345521 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.345526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.345766 (+ 240us) server_negotiation.cc:407] Connection header received
0810 06:24:43.345948 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.345951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.346000 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.346072 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.347001 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.347479 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.348407 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.348616 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.348685 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.350988 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:43.350992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:43.350994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:43.351016 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:43.352418 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.352943 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.352945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.352946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.353009 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.353333 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.353336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.353337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.353515 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:43.353601 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:43.354068 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 06:24:43.354185 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 06:24:43.355110 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.345288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:38005 (local address 127.0.0.1:44966)
0810 06:24:43.345652 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.345698 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:43.345844 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:43.346231 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:43.346234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:43.346605 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:43.346854 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:43.346868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.347573 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.347576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:43.348300 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:43.348309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.348741 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.348745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:43.348878 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.349373 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:43.349389 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:43.350849 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:43.352519 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:43.352527 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:43.352535 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:43.352863 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:43.353096 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:43.353100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:43.353102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:43.353244 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:43.353617 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:43.353623 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:43.353844 (+ 221us) client_negotiation.cc:769] Sending connection context
0810 06:24:43.353983 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 06:24:43.354308 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":64,"thread_start_us":116,"threads_started":1}
I20260810 06:24:43.355113 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.343688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43858 (local address 127.31.204.252:43047)
0810 06:24:43.344101 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.344119 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.344161 (+ 42us) server_negotiation.cc:407] Connection header received
0810 06:24:43.344971 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.344991 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.345301 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.345621 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.346470 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.347574 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.348366 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.348803 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.348898 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.351409 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:43.351415 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:43.351425 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:43.351446 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:43.352923 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.353288 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.353294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.353297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.353369 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.353604 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.353607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.353609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.353850 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:43.354049 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:43.354274 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 06:24:43.354399 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":266,"thread_start_us":104,"threads_started":1}
I20260810 06:24:43.356233 5047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:24:43 dist-test-slave-q9h9 krb5kdc[4648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343083, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.252@KRBTEST.COM for kudu/127.31.204.253@KRBTEST.COM
Aug 10 06:24:43 dist-test-slave-q9h9 krb5kdc[4648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:24:43.364457 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.356647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:39423 (local address 127.0.0.1:58710)
0810 06:24:43.356948 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.356961 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:43.357040 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:43.357393 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:43.357396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:43.357553 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:43.357657 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:43.357664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.358571 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.358574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:43.359163 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:43.359171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.359721 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.359724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:43.359816 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.360238 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:43.360248 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:43.361497 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:43.363159 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:43.363162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:43.363164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:43.363358 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:43.363604 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:43.363607 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:43.363609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:43.363684 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:43.364020 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:43.364023 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:43.364109 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 06:24:43.364190 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 06:24:43.364315 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
I20260810 06:24:43.364517 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.356759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.31.204.253:39423)
0810 06:24:43.356893 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.356897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.357012 (+ 115us) server_negotiation.cc:407] Connection header received
0810 06:24:43.357140 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.357143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.357196 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.357275 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.357778 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.358477 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.359261 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.359615 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.359717 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.361625 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:43.361628 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:43.361630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:43.361651 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:43.363062 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.363444 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.363447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.363449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.363525 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.363764 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.363767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.363769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.363934 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:43.364016 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:43.364255 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 06:24:43.364361 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:24:43.365193 5047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:24:43.365460 4803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.357316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.31.204.254:38005)
0810 06:24:43.357453 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.357456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.357469 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:24:43.357504 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.357506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.357545 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.357619 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.358260 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.358813 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.359648 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.359947 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.360032 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.362291 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:43.362295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:43.362297 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:43.362320 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:43.363461 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.363827 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.363830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.363832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.363907 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.364147 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.364150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.364152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.364322 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:43.365131 (+ 809us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:43.365222 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 06:24:43.365325 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 06:24:43.368090 4797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } attempt: 3
Aug 10 06:24:43 dist-test-slave-q9h9 krb5kdc[4648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343083, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.252@KRBTEST.COM for kudu/127.31.204.252@KRBTEST.COM
I20260810 06:24:43.370906 4924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } attempt: 3
Aug 10 06:24:43 dist-test-slave-q9h9 krb5kdc[4648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.253@KRBTEST.COM
I20260810 06:24:43.372599 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.365587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43047 (local address 127.0.0.1:43864)
0810 06:24:43.365667 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.365679 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:43.365755 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:43.365950 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:43.365953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:43.366124 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:43.366199 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:43.366205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.367044 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.367047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:43.367516 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:43.367523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.367927 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.367931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:43.367997 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.368401 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:43.368415 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:43.369637 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:43.371196 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:43.371199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:43.371201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:43.371424 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:43.371690 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:43.371692 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:43.371694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:43.371790 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:43.372191 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:43.372193 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:43.372266 (+ 73us) client_negotiation.cc:769] Sending connection context
0810 06:24:43.372355 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 06:24:43.372463 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260810 06:24:43.373016 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.365711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43864 (local address 127.31.204.252:43047)
0810 06:24:43.365798 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.365801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.365812 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:24:43.365837 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.365839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.365870 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.365930 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.366301 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.366954 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.367610 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.367840 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.367918 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.369758 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:43.369761 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:43.369764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:43.369786 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:43.371101 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.371515 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.371518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.371519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.371601 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.371876 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.371879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.371881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.372078 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:43.372160 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:43.372787 (+ 627us) server_negotiation.cc:299] Negotiation successful
0810 06:24:43.372882 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":35}
Aug 10 06:24:43 dist-test-slave-q9h9 krb5kdc[4648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343082, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for kudu/127.31.204.252@KRBTEST.COM
I20260810 06:24:43.376194 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90: Bootstrap starting.
Aug 10 06:24:43 dist-test-slave-q9h9 krb5kdc[4648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343083, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.253@KRBTEST.COM for kudu/127.31.204.252@KRBTEST.COM
I20260810 06:24:43.376820 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.366456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58716 (local address 127.31.204.253:39423)
0810 06:24:43.366547 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.366551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.366564 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:24:43.366643 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.366645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.366675 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.366738 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.367282 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.367733 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.368541 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.368827 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.368896 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.372080 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:43.372083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:43.372086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:43.372111 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:43.374760 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.375277 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.375280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.375282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.375365 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.375687 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.375690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.375691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.376043 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:43.376217 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:43.376565 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 06:24:43.376686 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 06:24:43.377645 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.368439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43047 (local address 127.0.0.1:43870)
0810 06:24:43.368597 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.368609 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:43.368679 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:43.369120 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:43.369124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:43.369285 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:43.369374 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:43.369381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.370812 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.370815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:43.371551 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:43.371562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.372147 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.372151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:43.372239 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.372617 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:43.372630 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:43.373916 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:43.375799 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:43.375802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:43.375803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:43.376024 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:43.376403 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:43.376406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:43.376408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:43.376500 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:43.377204 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:43.377207 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:43.377304 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 06:24:43.377383 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 06:24:43.377506 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260810 06:24:43.378688 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.368510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43870 (local address 127.31.204.252:43047)
0810 06:24:43.368900 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.368913 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.368925 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:24:43.368954 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.368957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.368994 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.369076 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.369481 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.370695 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.371669 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.372021 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.372409 (+ 388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.374057 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:43.374061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:43.374063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:43.374087 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:43.375692 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.376162 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.376164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.376166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.376250 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.376662 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.376665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.376667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.377074 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:43.378319 (+ 1245us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:43.378389 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 06:24:43.378542 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":113,"threads_started":1}
I20260810 06:24:43.379153 5047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:43.380084 5047 log.cc:824] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:43.380246 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:24:43.380661 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.371295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:43047 (local address 127.0.0.1:43882)
0810 06:24:43.371448 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:43.371460 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:43.371520 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:43.371993 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:43.371997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:43.372159 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:43.372261 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:43.372268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.373691 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.373695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:43.374419 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:43.374431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.375035 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.375039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:43.375120 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.375534 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:43.375548 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:43.376833 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:43.378832 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:43.378835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:43.378837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:43.379066 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:43.379750 (+ 684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:43.379753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:43.379755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:43.379842 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:43.380268 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:43.380271 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:43.380361 (+ 90us) client_negotiation.cc:769] Sending connection context
0810 06:24:43.380444 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 06:24:43.380563 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260810 06:24:43.381101 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.371329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43882 (local address 127.31.204.252:43047)
0810 06:24:43.371779 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.371787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.371798 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:24:43.371824 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.371827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.371864 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.372365 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.372471 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.373436 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.374558 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.374904 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.376955 (+ 2051us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.377048 (+ 93us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:43.377051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:43.377053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:43.377069 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:43.378713 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.379556 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.379558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.379560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.379638 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.379956 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.379958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.379960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.380141 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:43.380805 (+ 664us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:43.380890 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 06:24:43.380993 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":77,"threads_started":1}
I20260810 06:24:43.381822 4797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20: Bootstrap starting.
I20260810 06:24:43.383761 4924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9: Bootstrap starting.
I20260810 06:24:43.384305 4797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:43.385105 4797 log.cc:824] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:43.386181 4924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:43.386075 4752 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "dbfb0a8896894ce88304f04ab545be20"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44970
I20260810 06:24:43.386969 4924 log.cc:824] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:43.387590 4797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20: No bootstrap required, opened a new log
I20260810 06:24:43.388293 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90: No bootstrap required, opened a new log
I20260810 06:24:43.388919 4924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9: No bootstrap required, opened a new log
I20260810 06:24:43.390522 4797 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfb0a8896894ce88304f04ab545be20" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } } peers { permanent_uuid: "86a3314009874036ab6cd2864681b3b9" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } } peers { permanent_uuid: "95bf645f7e364ebdb0c6301d1ef6db90" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } }
I20260810 06:24:43.390754 4797 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbfb0a8896894ce88304f04ab545be20, State: Initialized, Role: FOLLOWER
I20260810 06:24:43.391070 4797 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfb0a8896894ce88304f04ab545be20" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } } peers { permanent_uuid: "86a3314009874036ab6cd2864681b3b9" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } } peers { permanent_uuid: "95bf645f7e364ebdb0c6301d1ef6db90" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } }
I20260810 06:24:43.392382 4797 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:43.392438 4752 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 06:24:43.392546 4752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:43.392609 4924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfb0a8896894ce88304f04ab545be20" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } } peers { permanent_uuid: "86a3314009874036ab6cd2864681b3b9" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } } peers { permanent_uuid: "95bf645f7e364ebdb0c6301d1ef6db90" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } }
I20260810 06:24:43.392618 5047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfb0a8896894ce88304f04ab545be20" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } } peers { permanent_uuid: "86a3314009874036ab6cd2864681b3b9" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } } peers { permanent_uuid: "95bf645f7e364ebdb0c6301d1ef6db90" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } }
I20260810 06:24:43.392802 4924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a3314009874036ab6cd2864681b3b9, State: Initialized, Role: FOLLOWER
I20260810 06:24:43.392833 5047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95bf645f7e364ebdb0c6301d1ef6db90, State: Initialized, Role: FOLLOWER
I20260810 06:24:43.393079 4924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfb0a8896894ce88304f04ab545be20" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } } peers { permanent_uuid: "86a3314009874036ab6cd2864681b3b9" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } } peers { permanent_uuid: "95bf645f7e364ebdb0c6301d1ef6db90" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } }
I20260810 06:24:43.393111 5047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfb0a8896894ce88304f04ab545be20" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } } peers { permanent_uuid: "86a3314009874036ab6cd2864681b3b9" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } } peers { permanent_uuid: "95bf645f7e364ebdb0c6301d1ef6db90" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } }
I20260810 06:24:43.393705 4752 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfb0a8896894ce88304f04ab545be20" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } } peers { permanent_uuid: "86a3314009874036ab6cd2864681b3b9" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } } peers { permanent_uuid: "95bf645f7e364ebdb0c6301d1ef6db90" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } }
I20260810 06:24:43.394356 5047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:43.394378 4924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:43.395545 5010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbfb0a8896894ce88304f04ab545be20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "95bf645f7e364ebdb0c6301d1ef6db90"
I20260810 06:24:43.395733 5010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:43.396984 5010 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbfb0a8896894ce88304f04ab545be20 in term 1.
I20260810 06:24:43.399237 4752 leader_election.cc:288] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [CANDIDATE]: Term 1 election: Requested vote from peers 86a3314009874036ab6cd2864681b3b9 (127.31.204.253:39423), 95bf645f7e364ebdb0c6301d1ef6db90 (127.31.204.252:43047)
W20260810 06:24:43.399266 5124 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:24:43.399333 5124 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:24:43.399845 5070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfb0a8896894ce88304f04ab545be20" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } } peers { permanent_uuid: "86a3314009874036ab6cd2864681b3b9" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } } peers { permanent_uuid: "95bf645f7e364ebdb0c6301d1ef6db90" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } } }
I20260810 06:24:43.399966 5070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 06:24:43.397823 5081 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 06:24:43.398062 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfb0a8896894ce88304f04ab545be20" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } } peers { permanent_uuid: "86a3314009874036ab6cd2864681b3b9" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } } peers { permanent_uuid: "95bf645f7e364ebdb0c6301d1ef6db90" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } } }
I20260810 06:24:43.398236 4706 leader_election.cc:302] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95bf645f7e364ebdb0c6301d1ef6db90, dbfb0a8896894ce88304f04ab545be20; no voters:
W20260810 06:24:43.402391 4718 tablet.cc:2369] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:24:43.403533 4938 master_runner.cc:232] Existing masters have differing terms: 1 vs 0
I20260810 06:24:43.403647 4938 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 06:24:43.404675 4807 master_runner.cc:263] No leader master found from master 86a3314009874036ab6cd2864681b3b9
I20260810 06:24:43.404733 4807 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 06:24:43.404963 5136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:43.405231 5136 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [term 1 LEADER]: Becoming Leader. State: Replica: dbfb0a8896894ce88304f04ab545be20, State: Running, Role: LEADER
I20260810 06:24:43.405607 5136 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfb0a8896894ce88304f04ab545be20" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } } peers { permanent_uuid: "86a3314009874036ab6cd2864681b3b9" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } } peers { permanent_uuid: "95bf645f7e364ebdb0c6301d1ef6db90" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } }
I20260810 06:24:43.406435 4675 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260810 06:24:43.406535 4675 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 06:24:43.406620 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 06:24:43.406759 5081 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:24:43.397145 5066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfb0a8896894ce88304f04ab545be20" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } } peers { permanent_uuid: "86a3314009874036ab6cd2864681b3b9" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } } peers { permanent_uuid: "95bf645f7e364ebdb0c6301d1ef6db90" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } } }
I20260810 06:24:43.397397 4868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbfb0a8896894ce88304f04ab545be20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "86a3314009874036ab6cd2864681b3b9"
I20260810 06:24:43.406953 4868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:43.408084 4868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbfb0a8896894ce88304f04ab545be20 in term 1.
I20260810 06:24:43.408699 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbfb0a8896894ce88304f04ab545be20. Latest consensus state: current_term: 1 leader_uuid: "dbfb0a8896894ce88304f04ab545be20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfb0a8896894ce88304f04ab545be20" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } } peers { permanent_uuid: "86a3314009874036ab6cd2864681b3b9" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } } peers { permanent_uuid: "95bf645f7e364ebdb0c6301d1ef6db90" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } } }
I20260810 06:24:43.408795 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:43.409000 5066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:24:43.409222 5142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:43.410177 5142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:43.416347 5010 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90 [term 1 FOLLOWER]: Refusing update from remote peer dbfb0a8896894ce88304f04ab545be20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260810 06:24:43.416618 5129 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:24:43.416702 5129 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:24:43.417013 5139 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95bf645f7e364ebdb0c6301d1ef6db90" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:24:43.420629 5070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbfb0a8896894ce88304f04ab545be20. Latest consensus state: current_term: 1 leader_uuid: "dbfb0a8896894ce88304f04ab545be20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfb0a8896894ce88304f04ab545be20" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } } peers { permanent_uuid: "86a3314009874036ab6cd2864681b3b9" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } } peers { permanent_uuid: "95bf645f7e364ebdb0c6301d1ef6db90" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } } }
I20260810 06:24:43.420748 5070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:24:43.421161 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.410366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.31.204.253:39423)
0810 06:24:43.410478 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.410483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.411031 (+ 548us) server_negotiation.cc:407] Connection header received
0810 06:24:43.411215 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.411219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.411265 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.411313 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.412326 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.413086 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.413907 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.414255 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.414349 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.416120 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:43.416123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:43.416126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:43.416151 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:43.417993 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.418495 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.418498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.418500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.418587 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.418878 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.418881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.418883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.419084 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:43.419200 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:43.420912 (+ 1712us) server_negotiation.cc:299] Negotiation successful
0810 06:24:43.421023 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:24:43.421612 4868 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9 [term 1 FOLLOWER]: Refusing update from remote peer dbfb0a8896894ce88304f04ab545be20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:24:43.422154 5139 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a3314009874036ab6cd2864681b3b9" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:24:43.425390 5066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbfb0a8896894ce88304f04ab545be20. Latest consensus state: current_term: 1 leader_uuid: "dbfb0a8896894ce88304f04ab545be20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfb0a8896894ce88304f04ab545be20" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } } peers { permanent_uuid: "86a3314009874036ab6cd2864681b3b9" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } } peers { permanent_uuid: "95bf645f7e364ebdb0c6301d1ef6db90" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } } }
I20260810 06:24:43.425505 5066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:24:43.427646 4803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.411182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.31.204.254:38005)
0810 06:24:43.411282 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.411287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.418266 (+ 6979us) server_negotiation.cc:407] Connection header received
0810 06:24:43.418426 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.418429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.418470 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.418516 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.419299 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.420049 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.421240 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.421577 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.421645 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.424079 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:43.424083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:43.424085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:43.424111 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:43.425652 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.426388 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.426392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.426394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.426477 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.426763 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.426766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.426768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.426966 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:43.427067 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:43.427386 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 06:24:43.427499 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 06:24:43.429100 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:43.411759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.31.204.252:43047)
0810 06:24:43.411867 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.411871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.419809 (+ 7938us) server_negotiation.cc:407] Connection header received
0810 06:24:43.420062 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.420065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.420105 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.420150 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.421698 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.422377 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.423446 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:43.423776 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:43.423850 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:43.425527 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:43.425531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:43.425534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:43.425559 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:43.427130 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.427599 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.427602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.427604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.427691 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:43.428166 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:43.428170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:43.428172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:43.428384 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:43.428504 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:43.428860 (+ 356us) server_negotiation.cc:299] Negotiation successful
0810 06:24:43.428971 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:24:43.430472 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dbfb0a8896894ce88304f04ab545be20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfb0a8896894ce88304f04ab545be20" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } } peers { permanent_uuid: "86a3314009874036ab6cd2864681b3b9" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } } peers { permanent_uuid: "95bf645f7e364ebdb0c6301d1ef6db90" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } } }
I20260810 06:24:43.430584 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:43.430809 5146 mvcc.cc:205] Tried to move back new op lower bound from 7316861269666131968 to 7316861269633527808. Current Snapshot: MvccSnapshot[applied={T|T < 7316861269666131968}]
I20260810 06:24:43.431439 5066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dbfb0a8896894ce88304f04ab545be20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfb0a8896894ce88304f04ab545be20" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } } peers { permanent_uuid: "86a3314009874036ab6cd2864681b3b9" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } } peers { permanent_uuid: "95bf645f7e364ebdb0c6301d1ef6db90" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } } }
I20260810 06:24:43.431540 5066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:24:43.432611 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dbfb0a8896894ce88304f04ab545be20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfb0a8896894ce88304f04ab545be20" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } } peers { permanent_uuid: "86a3314009874036ab6cd2864681b3b9" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } } peers { permanent_uuid: "95bf645f7e364ebdb0c6301d1ef6db90" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } } }
I20260810 06:24:43.432713 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:43.433297 5070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dbfb0a8896894ce88304f04ab545be20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfb0a8896894ce88304f04ab545be20" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 38005 } } peers { permanent_uuid: "86a3314009874036ab6cd2864681b3b9" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 39423 } } peers { permanent_uuid: "95bf645f7e364ebdb0c6301d1ef6db90" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 43047 } } }
I20260810 06:24:43.433398 5070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:24:43.434744 5151 mvcc.cc:205] Tried to move back new op lower bound from 7316861269666131968 to 7316861269633527808. Current Snapshot: MvccSnapshot[applied={T|T < 7316861269666131968}]
I20260810 06:24:43.437021 5142 catalog_manager.cc:1347] Generated new cluster ID: 07070b28ff874996822f622824e51fc4
I20260810 06:24:43.437076 5142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:43.477506 5142 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:43.478744 5142 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:43.493728 5142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbfb0a8896894ce88304f04ab545be20: Generated new TSK 0
I20260810 06:24:43.494642 5142 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 06:24:43.520087 4936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:43.518212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.31.204.253:39423)
0810 06:24:43.518346 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.518350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.518991 (+ 641us) server_negotiation.cc:407] Connection header received
0810 06:24:43.519210 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.519213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.519264 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.519318 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.519949 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58734: BlockingRecv error: recv got EOF from 127.0.0.1:58734 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 06:24:43.522208 4803 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:43.520737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.31.204.254:38005)
0810 06:24:43.520845 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.520849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.521319 (+ 470us) server_negotiation.cc:407] Connection header received
0810 06:24:43.521483 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.521485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.521534 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.521582 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.522090 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44984: BlockingRecv error: recv got EOF from 127.0.0.1:44984 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 06:24:43.523685 5055 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:43.521257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43892 (local address 127.31.204.252:43047)
0810 06:24:43.521384 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.521388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.522794 (+ 1406us) server_negotiation.cc:407] Connection header received
0810 06:24:43.522959 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.522962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.523009 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.523056 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.523566 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43892: BlockingRecv error: recv got EOF from 127.0.0.1:43892 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 06:24:43.527750 4936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:43.526252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.31.204.253:39423)
0810 06:24:43.526345 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.526348 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.526821 (+ 473us) server_negotiation.cc:407] Connection header received
0810 06:24:43.527032 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.527035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.527107 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.527172 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.527656 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58748: BlockingRecv error: recv got EOF from 127.0.0.1:58748 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 06:24:43.532459 4936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:43.531008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58750 (local address 127.31.204.253:39423)
0810 06:24:43.531101 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.531105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.531633 (+ 528us) server_negotiation.cc:407] Connection header received
0810 06:24:43.531807 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.531810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.531857 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.531904 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.532370 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58750: BlockingRecv error: recv got EOF from 127.0.0.1:58750 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 06:24:43.537708 4936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:43.536303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58760 (local address 127.31.204.253:39423)
0810 06:24:43.536397 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.536401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.536894 (+ 493us) server_negotiation.cc:407] Connection header received
0810 06:24:43.537069 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.537072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.537116 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.537175 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.537621 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58760: BlockingRecv error: recv got EOF from 127.0.0.1:58760 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 06:24:43.545089 4936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:43.543566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58768 (local address 127.31.204.253:39423)
0810 06:24:43.543700 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.543704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.544240 (+ 536us) server_negotiation.cc:407] Connection header received
0810 06:24:43.544423 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.544426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.544471 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.544521 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.545001 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58768: BlockingRecv error: recv got EOF from 127.0.0.1:58768 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 06:24:43.556569 4936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:43.555026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58776 (local address 127.31.204.253:39423)
0810 06:24:43.555124 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.555128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.555671 (+ 543us) server_negotiation.cc:407] Connection header received
0810 06:24:43.555866 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.555869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.555919 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.555971 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.556479 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58776: BlockingRecv error: recv got EOF from 127.0.0.1:58776 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 06:24:43.576372 4936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:43.574736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58778 (local address 127.31.204.253:39423)
0810 06:24:43.574865 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.574869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.575457 (+ 588us) server_negotiation.cc:407] Connection header received
0810 06:24:43.575641 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.575644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.575695 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.575744 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.576264 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58778: BlockingRecv error: recv got EOF from 127.0.0.1:58778 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 06:24:43.611716 4936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:43.610501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.31.204.253:39423)
0810 06:24:43.610628 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.610632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.610892 (+ 260us) server_negotiation.cc:407] Connection header received
0810 06:24:43.611027 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.611030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.611082 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.611148 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.611583 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58780: BlockingRecv error: recv got EOF from 127.0.0.1:58780 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 06:24:43.678920 4936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:43.677769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58790 (local address 127.31.204.253:39423)
0810 06:24:43.677920 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.677925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.678062 (+ 137us) server_negotiation.cc:407] Connection header received
0810 06:24:43.678244 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.678247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.678296 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.678365 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.678813 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58790: BlockingRecv error: recv got EOF from 127.0.0.1:58790 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260810 06:24:43.810138 4936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:43.808985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58802 (local address 127.31.204.253:39423)
0810 06:24:43.809128 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:43.809132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:43.809336 (+ 204us) server_negotiation.cc:407] Connection header received
0810 06:24:43.809463 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:43.809466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:43.809515 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:43.809592 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:43.809983 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58802: BlockingRecv error: recv got EOF from 127.0.0.1:58802 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 06:24:44.069358 4936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:44.068371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.31.204.253:39423)
0810 06:24:44.068531 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:44.068534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:44.068553 (+ 19us) server_negotiation.cc:407] Connection header received
0810 06:24:44.068690 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:44.068692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:44.068742 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:44.068814 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:44.069212 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58804: BlockingRecv error: recv got EOF from 127.0.0.1:58804 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 06:24:44.400610 5124 catalog_manager.cc:1259] Loaded cluster ID: 07070b28ff874996822f622824e51fc4
I20260810 06:24:44.400691 5124 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9: loading cluster ID for follower catalog manager: success
I20260810 06:24:44.402361 5124 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9: acquiring CA information for follower catalog manager: success
I20260810 06:24:44.403537 5124 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 86a3314009874036ab6cd2864681b3b9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 06:24:44.417800 5129 catalog_manager.cc:1259] Loaded cluster ID: 07070b28ff874996822f622824e51fc4
I20260810 06:24:44.417863 5129 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90: loading cluster ID for follower catalog manager: success
I20260810 06:24:44.419459 5129 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90: acquiring CA information for follower catalog manager: success
I20260810 06:24:44.420538 5129 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 95bf645f7e364ebdb0c6301d1ef6db90: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 06:24:44.587798 5269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:44.583426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.31.204.253:39423)
0810 06:24:44.583690 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:44.583695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:44.583712 (+ 17us) server_negotiation.cc:407] Connection header received
0810 06:24:44.583847 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:44.583850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:44.584020 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:44.584155 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:24:44.584368 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:44.585182 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:44.586092 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:44.586454 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:44.586560 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:44.586816 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:24:44.587398 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:24:44.587520 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:44.587550 (+ 30us) server_negotiation.cc:299] Negotiation successful
0810 06:24:44.587612 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":108,"threads_started":1}
I20260810 06:24:44.594774 5275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:44.590841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43902 (local address 127.31.204.252:43047)
0810 06:24:44.591069 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:44.591073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:44.591173 (+ 100us) server_negotiation.cc:407] Connection header received
0810 06:24:44.591346 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:44.591349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:44.591461 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:44.591558 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:24:44.591764 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:44.592256 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:44.593078 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:44.593417 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:44.593520 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:44.593829 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:24:44.594335 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:24:44.594450 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:44.594567 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 06:24:44.594626 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":70,"threads_started":1}
I20260810 06:24:44.602711 5275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:44.598431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43914 (local address 127.31.204.252:43047)
0810 06:24:44.598536 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:44.598540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:44.598857 (+ 317us) server_negotiation.cc:407] Connection header received
0810 06:24:44.598939 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:44.598942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:44.598982 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:44.599055 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:24:44.599292 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:44.600052 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:44.601377 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:44.601715 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:44.601786 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:44.602184 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:24:44.602262 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:24:44.602306 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:44.602519 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 06:24:44.602570 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 06:24:44.603000 5282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:44.598054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.31.204.254:38005)
0810 06:24:44.598812 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:44.598816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:44.598830 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:24:44.598918 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:44.598921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:44.599073 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:44.599182 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:24:44.599434 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:44.600194 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:44.601406 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:44.601749 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:44.601847 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:44.602252 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:24:44.602543 (+ 291us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:24:44.602614 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:44.602799 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 06:24:44.602861 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":63,"threads_started":1}
I20260810 06:24:44.603021 5269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:44.598068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58818 (local address 127.31.204.253:39423)
0810 06:24:44.598186 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:44.598190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:44.599453 (+ 1263us) server_negotiation.cc:407] Connection header received
0810 06:24:44.599612 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:44.599615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:44.599662 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:44.599705 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:24:44.600035 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:44.600750 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:44.601848 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:44.602180 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:44.602247 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:44.602603 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:24:44.602676 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:24:44.602718 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:44.602871 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 06:24:44.602918 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:24:44.606185 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 4675
I20260810 06:24:44.619553 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 4807
I20260810 06:24:44.631309 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 4938
2026-08-10T06:24:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3169 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:44 dist-test-slave-q9h9 krb5kdc[5292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:44 dist-test-slave-q9h9 krb5kdc[5292](info): set up 2 sockets
Aug 10 06:24:44 dist-test-slave-q9h9 krb5kdc[5292](info): commencing operation
krb5kdc: starting...
W20260810 06:24:45.796820 32563 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/test-user.keytab.
Aug 10 06:24:45 dist-test-slave-q9h9 krb5kdc[5292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.204.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
Entry for principal kudu/127.31.204.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab.
I20260810 06:24:45.909531 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36761
--webserver_interface=127.31.204.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:46503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:36761,127.31.204.253:44099,127.31.204.252:42169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:46.018296 5319 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 06:24:46.018543 5319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:46.018586 5319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:46.018647 5319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:46.022816 5319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:46.022881 5319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:46.022907 5319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:46.022930 5319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:46.022959 5319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:46.028137 5319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:46503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:36761,127.31.204.253:44099,127.31.204.252:42169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.254.keytab
--principal=kudu/127.31.204.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.254:36761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.204.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:46.029110 5319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:46.030140 5319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:46.038178 5328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:46.038273 5319 server_base.cc:1034] running on GCE node
W20260810 06:24:46.038178 5329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:46.038184 5331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:46.038961 5319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:46.039975 5319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:46.041118 5319 hybrid_clock.cc:648] HybridClock initialized: now 1786343086041101 us; error 38 us; skew 500 ppm
Aug 10 06:24:46 dist-test-slave-q9h9 krb5kdc[5292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343086, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:46.043654 5319 init.cc:377] Logged in from keytab as kudu/127.31.204.254@KRBTEST.COM (short username kudu)
I20260810 06:24:46.044454 5319 webserver.cc:460] Webserver started at http://127.31.204.254:42151/ using document root <none> and password file <none>
I20260810 06:24:46.044965 5319 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:46.045032 5319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:46.045228 5319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:46.047230 5319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-0/data/instance:
uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e"
format_stamp: "Formatted at 2026-08-10 06:24:46 on dist-test-slave-q9h9"
server_key: "9632541608561600efbcb39f3cbd38fa"
server_key_iv: "67001fe45450f579a91fe31995169a0e"
server_key_version: "encryptionkey@0"
I20260810 06:24:46.047715 5319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-0/wal/instance:
uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e"
format_stamp: "Formatted at 2026-08-10 06:24:46 on dist-test-slave-q9h9"
server_key: "9632541608561600efbcb39f3cbd38fa"
server_key_iv: "67001fe45450f579a91fe31995169a0e"
server_key_version: "encryptionkey@0"
I20260810 06:24:46.051064 5319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:24:46.053242 5349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:46.054382 5319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:24:46.054492 5319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-0/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-0/wal
uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e"
format_stamp: "Formatted at 2026-08-10 06:24:46 on dist-test-slave-q9h9"
server_key: "9632541608561600efbcb39f3cbd38fa"
server_key_iv: "67001fe45450f579a91fe31995169a0e"
server_key_version: "encryptionkey@0"
I20260810 06:24:46.054576 5319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:46.071285 5319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:46.073952 5319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:46.074137 5319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:46.080885 5319 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.254:36761
I20260810 06:24:46.081125 5433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.254:36761 every 8 connection(s)
I20260810 06:24:46.081779 5319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:46.084430 5436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:46.084875 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 5319
I20260810 06:24:46.084990 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-0/wal/instance
I20260810 06:24:46.085286 32563 external_mini_cluster.cc:1428] Setting key bc187e3c227c3c2ac59699b5169712d0
I20260810 06:24:46.088620 5436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.204.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.253@KRBTEST.COM" created.
Aug 10 06:24:46 dist-test-slave-q9h9 krb5kdc[5292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343086, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:24:46.103441 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.090917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36761 (local address 127.0.0.1:35578)
0810 06:24:46.091412 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.091453 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:46.092331 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:46.092967 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:46.092973 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:46.093379 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:46.093603 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:46.093616 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.094871 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.094875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:46.095583 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:46.095595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.096412 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.096416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:46.096527 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.097048 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:46.097063 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:46.098690 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:46.100738 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:46.100746 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:46.100754 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:46.100965 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:46.101348 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:46.101351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:46.101353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:46.101468 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:46.102199 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:46.102204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:46.102320 (+ 116us) client_negotiation.cc:769] Sending connection context
0810 06:24:46.102450 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 06:24:46.102683 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":142,"threads_started":1}
I20260810 06:24:46.103446 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.091882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35578 (local address 127.31.204.254:36761)
0810 06:24:46.092162 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.092172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.092217 (+ 45us) server_negotiation.cc:407] Connection header received
0810 06:24:46.092601 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.092605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.092785 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.092943 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.093743 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.094753 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.095711 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.096244 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.096409 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.098855 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:46.098861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:46.098870 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:46.098893 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:46.100633 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.101072 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.101078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.101079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.101243 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.101556 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.101559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.101561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.102072 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:46.102225 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:46.102558 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 06:24:46.102785 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":182,"thread_start_us":89,"threads_started":1}
I20260810 06:24:46.105767 5436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:24:46.106493 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.106213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:44099 (local address 127.0.0.1:58144)
0810 06:24:46.106373 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.106440 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.253:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 06:24:46.106658 5353 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.204.253:44099: connect: Connection refused (error 111)
I20260810 06:24:46.107530 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.107293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:44099 (local address 127.0.0.1:58156)
0810 06:24:46.107433 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.107482 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.253:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260810 06:24:46.107649 5436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.253:44099: Network error: Client connection negotiation failed: client connection to 127.31.204.253:44099: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.253.keytab.
Entry for principal kudu/127.31.204.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.253.keytab.
I20260810 06:24:46.111989 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.253:44099
--webserver_interface=127.31.204.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.253.keytab
--principal=kudu/127.31.204.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:46503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:36761,127.31.204.253:44099,127.31.204.252:42169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:24:46.156868 5436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } attempt: 1
I20260810 06:24:46.157748 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.157422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:44099 (local address 127.0.0.1:58166)
0810 06:24:46.157574 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.157635 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.253:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 06:24:46.158475 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.158250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:44099 (local address 127.0.0.1:58168)
0810 06:24:46.158358 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.158412 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.253:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 06:24:46.158622 5436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.253:44099: Network error: Client connection negotiation failed: client connection to 127.31.204.253:44099: connect: Connection refused (error 111)
W20260810 06:24:46.219712 5447 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 06:24:46.219985 5447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:46.220027 5447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:46.220088 5447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:46.224004 5447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:46.224053 5447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:46.224078 5447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:46.224099 5447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:46.224123 5447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:46.226776 5436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } attempt: 2
I20260810 06:24:46.227563 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.227229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:44099 (local address 127.0.0.1:58170)
0810 06:24:46.227398 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.227471 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.253:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 06:24:46.228199 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.228006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:44099 (local address 127.0.0.1:58178)
0810 06:24:46.228092 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.228140 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.253:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 06:24:46.228317 5436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.253:44099: Network error: Client connection negotiation failed: client connection to 127.31.204.253:44099: connect: Connection refused (error 111)
I20260810 06:24:46.229480 5447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:46503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:36761,127.31.204.253:44099,127.31.204.252:42169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.253.keytab
--principal=kudu/127.31.204.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.253:44099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.204.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:46.230446 5447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:46.231406 5447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:46.239486 5457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:46.239509 5461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:46.239651 5447 server_base.cc:1034] running on GCE node
W20260810 06:24:46.239804 5458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:46.240324 5447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:46.241335 5447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:46.242476 5447 hybrid_clock.cc:648] HybridClock initialized: now 1786343086242454 us; error 35 us; skew 500 ppm
Aug 10 06:24:46 dist-test-slave-q9h9 krb5kdc[5292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343086, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:46.244961 5447 init.cc:377] Logged in from keytab as kudu/127.31.204.253@KRBTEST.COM (short username kudu)
I20260810 06:24:46.245859 5447 webserver.cc:460] Webserver started at http://127.31.204.253:33655/ using document root <none> and password file <none>
I20260810 06:24:46.246377 5447 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:46.246435 5447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:46.246595 5447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:46.248479 5447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-1/data/instance:
uuid: "7f6e5871d8794034988627a01b1bc20f"
format_stamp: "Formatted at 2026-08-10 06:24:46 on dist-test-slave-q9h9"
server_key: "bad313e99a43b3f5a8f727641aa6fb8f"
server_key_iv: "42320f3737a05e4ba4c0e7e8d6a55798"
server_key_version: "encryptionkey@0"
I20260810 06:24:46.248976 5447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-1/wal/instance:
uuid: "7f6e5871d8794034988627a01b1bc20f"
format_stamp: "Formatted at 2026-08-10 06:24:46 on dist-test-slave-q9h9"
server_key: "bad313e99a43b3f5a8f727641aa6fb8f"
server_key_iv: "42320f3737a05e4ba4c0e7e8d6a55798"
server_key_version: "encryptionkey@0"
I20260810 06:24:46.252487 5447 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:24:46.254737 5473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:46.255914 5447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:46.256021 5447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-1/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-1/wal
uuid: "7f6e5871d8794034988627a01b1bc20f"
format_stamp: "Formatted at 2026-08-10 06:24:46 on dist-test-slave-q9h9"
server_key: "bad313e99a43b3f5a8f727641aa6fb8f"
server_key_iv: "42320f3737a05e4ba4c0e7e8d6a55798"
server_key_version: "encryptionkey@0"
I20260810 06:24:46.256127 5447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:46.275235 5447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:46.277768 5447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:46.277937 5447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:46.284538 5447 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.253:44099
I20260810 06:24:46.284786 5551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.253:44099 every 8 connection(s)
I20260810 06:24:46.285444 5447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-1/data/info.pb
I20260810 06:24:46.287406 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 5447
I20260810 06:24:46.287520 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-1/wal/instance
I20260810 06:24:46.287825 32563 external_mini_cluster.cc:1428] Setting key 90f939c3b06999df82dd0d4e308cd1a5
I20260810 06:24:46.288319 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:46.292246 5553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.204.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.204.252@KRBTEST.COM" created.
Aug 10 06:24:46 dist-test-slave-q9h9 krb5kdc[5292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343086, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.253@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:24:46.306113 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.294092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35594 (local address 127.31.204.254:36761)
0810 06:24:46.294246 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.294251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.295174 (+ 923us) server_negotiation.cc:407] Connection header received
0810 06:24:46.296372 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.296375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.296424 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.296498 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.297679 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.298238 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.299383 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.299701 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.299814 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.302111 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:46.302114 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:46.302116 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:46.302140 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:46.303582 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.304250 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.304254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.304256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.304356 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.304745 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.304749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.304751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.304985 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:46.305092 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:46.305776 (+ 684us) server_negotiation.cc:299] Negotiation successful
0810 06:24:46.305912 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 06:24:46.307236 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.294457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36761 (local address 127.0.0.1:35594)
0810 06:24:46.295002 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.295058 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:46.296138 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:46.296700 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:46.296719 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:46.297057 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:46.297548 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:46.297563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.298360 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.298363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:46.299273 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:46.299285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.299841 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.299844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:46.299982 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.300523 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:46.300538 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:46.301934 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:46.303720 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:46.303726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:46.303739 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:46.304132 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:46.304479 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:46.304482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:46.304484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:46.304647 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:46.305087 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:46.305093 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:46.305451 (+ 358us) client_negotiation.cc:769] Sending connection context
0810 06:24:46.305691 (+ 240us) client_negotiation.cc:240] Negotiation successful
0810 06:24:46.306054 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":122,"threads_started":1}
I20260810 06:24:46.308597 5553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.204.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.252.keytab.
Entry for principal kudu/127.31.204.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.252.keytab.
Aug 10 06:24:46 dist-test-slave-q9h9 krb5kdc[5292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343086, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.253@KRBTEST.COM for kudu/127.31.204.253@KRBTEST.COM
I20260810 06:24:46.314455 32563 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DS5L1/build/debug/bin/kudu
/tmp/dist-test-task7DS5L1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.252:42169
--webserver_interface=127.31.204.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.252.keytab
--principal=kudu/127.31.204.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.204.212:46503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.204.254:36761,127.31.204.253:44099,127.31.204.252:42169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:24:46.316996 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.309024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:44099 (local address 127.0.0.1:58190)
0810 06:24:46.309132 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.309165 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:46.309231 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:46.309815 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:46.309820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:46.309958 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:46.310063 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:46.310070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.311034 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.311038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:46.311586 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:46.311594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.312148 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.312152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:46.312278 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.312677 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:46.312690 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:46.313887 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:46.315639 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:46.315642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:46.315644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:46.315879 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:46.316165 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:46.316167 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:46.316168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:46.316233 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:46.316664 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:46.316666 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:46.316733 (+ 67us) client_negotiation.cc:769] Sending connection context
0810 06:24:46.316802 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 06:24:46.316890 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":16}
I20260810 06:24:46.317173 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.309129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.31.204.253:44099)
0810 06:24:46.309421 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.309430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.309446 (+ 16us) server_negotiation.cc:407] Connection header received
0810 06:24:46.309475 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.309479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.309614 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.309731 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.310171 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.310938 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.311682 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.312053 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.312141 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.314053 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:46.314063 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:46.314076 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:46.314098 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:46.315536 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.315990 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.315996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.315998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.316091 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.316308 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.316311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.316313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.316582 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:46.316708 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:46.316903 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 06:24:46.317043 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":213,"thread_start_us":120,"threads_started":1}
I20260810 06:24:46.318722 5553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:24:46.319363 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.319143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:42169 (local address 127.0.0.1:40578)
0810 06:24:46.319248 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.319313 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 06:24:46.319458 5436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } attempt: 3
W20260810 06:24:46.319741 5476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.204.252:42169: connect: Connection refused (error 111)
I20260810 06:24:46.320861 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.320623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:42169 (local address 127.0.0.1:40580)
0810 06:24:46.320752 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.320799 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260810 06:24:46.321205 5553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.252:42169: Network error: Client connection negotiation failed: client connection to 127.31.204.252:42169: connect: Connection refused (error 111)
Aug 10 06:24:46 dist-test-slave-q9h9 krb5kdc[5292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343086, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for kudu/127.31.204.253@KRBTEST.COM
I20260810 06:24:46.327262 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.319874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.31.204.253:44099)
0810 06:24:46.319966 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.319970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.320096 (+ 126us) server_negotiation.cc:407] Connection header received
0810 06:24:46.320197 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.320199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.320245 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.320318 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.321195 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.321712 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.322570 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.322796 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.322878 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.324604 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:46.324607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:46.324609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:46.324628 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:46.325831 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.326262 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.326265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.326266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.326332 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.326570 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.326573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.326574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.326723 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:46.326805 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:46.327041 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 06:24:46.327131 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 06:24:46.327263 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.319828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:44099 (local address 127.0.0.1:58202)
0810 06:24:46.320026 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.320040 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:46.320128 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:46.320351 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:46.320354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:46.320518 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:46.321087 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:46.321098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.321813 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.321816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:46.322464 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:46.322473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.322891 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.322894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:46.322987 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.323389 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:46.323398 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:46.324487 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:46.325927 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:46.325930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:46.325932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:46.326173 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:46.326412 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:46.326415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:46.326417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:46.326493 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:46.326810 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:46.326813 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:46.326902 (+ 89us) client_negotiation.cc:769] Sending connection context
0810 06:24:46.326985 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 06:24:46.327108 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260810 06:24:46.328342 5436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:24:46.328908 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.328713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:42169 (local address 127.0.0.1:40596)
0810 06:24:46.328808 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.328850 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 06:24:46.329504 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.329297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:42169 (local address 127.0.0.1:40604)
0810 06:24:46.329417 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.329460 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 06:24:46.329602 5436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.252:42169: Network error: Client connection negotiation failed: client connection to 127.31.204.252:42169: connect: Connection refused (error 111)
I20260810 06:24:46.360730 5436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } attempt: 1
I20260810 06:24:46.361286 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.361066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:42169 (local address 127.0.0.1:40616)
0810 06:24:46.361182 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.361230 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 06:24:46.361887 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.361726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:42169 (local address 127.0.0.1:40618)
0810 06:24:46.361803 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.361842 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 06:24:46.361996 5436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.252:42169: Network error: Client connection negotiation failed: client connection to 127.31.204.252:42169: connect: Connection refused (error 111)
I20260810 06:24:46.370426 5553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } attempt: 1
I20260810 06:24:46.370951 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.370740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:42169 (local address 127.0.0.1:40632)
0810 06:24:46.370850 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.370895 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 06:24:46.371537 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.371353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:42169 (local address 127.0.0.1:40638)
0810 06:24:46.371448 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.371490 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 06:24:46.371636 5553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.252:42169: Network error: Client connection negotiation failed: client connection to 127.31.204.252:42169: connect: Connection refused (error 111)
W20260810 06:24:46.422317 5562 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 06:24:46.422544 5562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:46.422587 5562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:46.422643 5562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:46.426486 5562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:46.426532 5562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:46.426556 5562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:46.426586 5562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:46.426616 5562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:46.431541 5562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.204.212:46503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.204.254:36761,127.31.204.253:44099,127.31.204.252:42169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/krb5kdc/kudu_127.31.204.252.keytab
--principal=kudu/127.31.204.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.204.252:42169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.204.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:46.432385 5562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:46.433265 5562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:24:46.437148 5436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } attempt: 2
I20260810 06:24:46.437840 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.437589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:42169 (local address 127.0.0.1:40648)
0810 06:24:46.437722 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.437775 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 06:24:46.438472 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.438280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:42169 (local address 127.0.0.1:40654)
0810 06:24:46.438376 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.438420 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 06:24:46.438585 5436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.252:42169: Network error: Client connection negotiation failed: client connection to 127.31.204.252:42169: connect: Connection refused (error 111)
I20260810 06:24:46.439764 5553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } attempt: 2
I20260810 06:24:46.440385 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.440108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:42169 (local address 127.0.0.1:40656)
0810 06:24:46.440204 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.440255 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 06:24:46.440994 5572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:46.441162 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.440923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:42169 (local address 127.0.0.1:40658)
0810 06:24:46.441050 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.441101 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.204.252:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 06:24:46.441282 5573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:46.441035 5575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:46.441304 5562 server_base.cc:1034] running on GCE node
W20260810 06:24:46.441385 5553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.204.252:42169: Network error: Client connection negotiation failed: client connection to 127.31.204.252:42169: connect: Connection refused (error 111)
I20260810 06:24:46.441826 5562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:46.442795 5562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:46.443936 5562 hybrid_clock.cc:648] HybridClock initialized: now 1786343086443916 us; error 32 us; skew 500 ppm
Aug 10 06:24:46 dist-test-slave-q9h9 krb5kdc[5292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343086, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:46.446365 5562 init.cc:377] Logged in from keytab as kudu/127.31.204.252@KRBTEST.COM (short username kudu)
I20260810 06:24:46.447101 5562 webserver.cc:460] Webserver started at http://127.31.204.252:35725/ using document root <none> and password file <none>
I20260810 06:24:46.447537 5562 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:46.447590 5562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:46.447750 5562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:46.449483 5562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-2/data/instance:
uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555"
format_stamp: "Formatted at 2026-08-10 06:24:46 on dist-test-slave-q9h9"
server_key: "31842834c1fd6e5abbd515d20b5bee4a"
server_key_iv: "a193d2555f00c3dd6fa50981fdf54e33"
server_key_version: "encryptionkey@0"
I20260810 06:24:46.449961 5562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-2/wal/instance:
uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555"
format_stamp: "Formatted at 2026-08-10 06:24:46 on dist-test-slave-q9h9"
server_key: "31842834c1fd6e5abbd515d20b5bee4a"
server_key_iv: "a193d2555f00c3dd6fa50981fdf54e33"
server_key_version: "encryptionkey@0"
I20260810 06:24:46.453186 5562 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:24:46.455339 5586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:46.456372 5562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:24:46.456477 5562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-2/data,/tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-2/wal
uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555"
format_stamp: "Formatted at 2026-08-10 06:24:46 on dist-test-slave-q9h9"
server_key: "31842834c1fd6e5abbd515d20b5bee4a"
server_key_iv: "a193d2555f00c3dd6fa50981fdf54e33"
server_key_version: "encryptionkey@0"
I20260810 06:24:46.456557 5562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:46.469343 5562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:46.471633 5562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:46.471792 5562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:46.477763 5562 rpc_server.cc:307] RPC server started. Bound to: 127.31.204.252:42169
I20260810 06:24:46.477993 5671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.204.252:42169 every 8 connection(s)
I20260810 06:24:46.478580 5562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-2/data/info.pb
I20260810 06:24:46.480522 32563 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DS5L1/build/debug/bin/kudu as pid 5562
I20260810 06:24:46.480635 32563 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DS5L1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786342803670820-32563-0/minicluster-data/master-2/wal/instance
I20260810 06:24:46.480892 32563 external_mini_cluster.cc:1428] Setting key 1bae021eebd7447091ff3ff82171c460
I20260810 06:24:46.481097 5673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:46.485662 5673 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:24:46 dist-test-slave-q9h9 krb5kdc[5292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.252@KRBTEST.COM
Aug 10 06:24:46 dist-test-slave-q9h9 krb5kdc[5292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343086, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.252@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:24:46.494194 5679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.482073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40666 (local address 127.31.204.252:42169)
0810 06:24:46.482473 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.482491 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.482529 (+ 38us) server_negotiation.cc:407] Connection header received
0810 06:24:46.483196 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.483217 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.483605 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.483952 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.484589 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.485994 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.486848 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.487339 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.487466 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.490164 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:46.490168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:46.490177 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:46.490202 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:46.491700 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.492099 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.492105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.492109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.492204 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.492452 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.492456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.492458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.492867 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:46.493031 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:46.493315 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 06:24:46.493545 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":107,"threads_started":1}
I20260810 06:24:46.495342 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.487150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.254:36761 (local address 127.0.0.1:35602)
0810 06:24:46.487537 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.487579 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:46.487692 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:46.487986 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:46.487991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:46.488340 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:46.488459 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:46.488465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.489346 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.489349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:46.490085 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:46.490094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.490509 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.490513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:46.490628 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.491005 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:46.491016 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:46.492257 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:46.493837 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:46.493845 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:46.493850 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:46.494093 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:46.494376 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:46.494379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:46.494381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:46.494469 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:46.494835 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:46.494840 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:46.494950 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 06:24:46.495088 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 06:24:46.495213 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":148,"threads_started":1}
I20260810 06:24:46.495428 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.487244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35602 (local address 127.31.204.254:36761)
0810 06:24:46.487368 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.487372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.487662 (+ 290us) server_negotiation.cc:407] Connection header received
0810 06:24:46.487781 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.487785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.487837 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.487924 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.488579 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.489225 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.490202 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.490415 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.490497 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.492383 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:46.492386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:46.492388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:46.492411 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:46.493745 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.494180 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.494184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.494185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.494265 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.494550 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.494553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.494554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.494729 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:46.494821 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:46.495174 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 06:24:46.495289 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:24:46.496454 5673 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:24:46 dist-test-slave-q9h9 krb5kdc[5292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343086, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.252@KRBTEST.COM for kudu/127.31.204.253@KRBTEST.COM
Aug 10 06:24:46 dist-test-slave-q9h9 krb5kdc[5292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.254@KRBTEST.COM
I20260810 06:24:46.503733 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.496830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.253:44099 (local address 127.0.0.1:58212)
0810 06:24:46.496962 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.496971 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:46.497034 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:46.497255 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:46.497258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:46.497392 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:46.497468 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:46.497472 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.498165 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.498169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:46.498639 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:46.498646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.499108 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.499111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:46.499206 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.499516 (+ 310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:46.499530 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:46.500639 (+ 1109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:46.502373 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:46.502376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:46.502377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:46.502553 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:46.502824 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:46.502826 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:46.502828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:46.502917 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:46.503358 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:46.503362 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:46.503449 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 06:24:46.503535 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 06:24:46.503622 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 06:24:46.503868 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.496949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.31.204.253:44099)
0810 06:24:46.497075 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.497079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.497092 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:24:46.497129 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.497132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.497171 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.497240 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.497571 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.498072 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.498747 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.499000 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.499095 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.500772 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:46.500775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:46.500777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:46.500800 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:46.502275 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.502642 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.502646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.502648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.502732 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.503035 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.503037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.503038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.503270 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:46.503356 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:46.503603 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 06:24:46.503715 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:24:46.504379 5673 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:24:46.504474 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.496279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35608 (local address 127.31.204.254:36761)
0810 06:24:46.496387 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.496390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.496401 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:24:46.496505 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.496507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.496541 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.496613 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.497195 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.497634 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.498399 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.498594 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.498673 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.501364 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:46.501367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:46.501370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:46.501395 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:46.502723 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.503136 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.503139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.503140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.503214 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.503476 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.503478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.503480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.503661 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:46.503986 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:46.504236 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 06:24:46.504357 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 06:24:46 dist-test-slave-q9h9 krb5kdc[5292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343086, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.252@KRBTEST.COM for kudu/127.31.204.252@KRBTEST.COM
Aug 10 06:24:46 dist-test-slave-q9h9 krb5kdc[5292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.204.253@KRBTEST.COM
I20260810 06:24:46.512117 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.504765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:42169 (local address 127.0.0.1:40678)
0810 06:24:46.504841 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.504851 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:46.504904 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:46.505323 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:46.505324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:46.505453 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:46.505522 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:46.505529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.506455 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.506458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:46.507220 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:46.507228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.507769 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.507772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:46.507840 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.508154 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:46.508167 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:46.509346 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:46.510855 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:46.510859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:46.510860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:46.511081 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:46.511334 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:46.511336 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:46.511338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:46.511428 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:46.511802 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:46.511804 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:46.511870 (+ 66us) client_negotiation.cc:769] Sending connection context
0810 06:24:46.511938 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 06:24:46.512020 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":2}
I20260810 06:24:46.512259 5679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.505008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.31.204.252:42169)
0810 06:24:46.505100 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.505103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.505114 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:24:46.505145 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.505148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.505183 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.505251 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.505630 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.506367 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.507312 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.507661 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.507767 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.509470 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:46.509472 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:46.509474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:46.509495 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:46.510743 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.511164 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.511167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.511169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.511252 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.511509 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.511512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.511514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.511702 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:46.511815 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:46.512001 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 06:24:46.512097 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":39}
I20260810 06:24:46.513525 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.505467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58216 (local address 127.31.204.253:44099)
0810 06:24:46.505569 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.505572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.505584 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:24:46.505613 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.505616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.505650 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.505708 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.506369 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.507060 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.507970 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.508230 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.508333 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.510578 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:46.510582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:46.510585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:46.510610 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:46.512076 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.512518 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.512521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.512522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.512592 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.512826 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.512829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.512831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.512977 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:46.513179 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:46.513305 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 06:24:46.513398 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:24:46.515225 5673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555: Bootstrap starting.
I20260810 06:24:46.516388 32563 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:24:46.517688 5673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:46.518381 5673 log.cc:824] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:46.517556 5390 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35608
I20260810 06:24:46.519989 5673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555: No bootstrap required, opened a new log
I20260810 06:24:46.522635 5673 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } } peers { permanent_uuid: "7f6e5871d8794034988627a01b1bc20f" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } } peers { permanent_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } }
I20260810 06:24:46.522810 5673 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ee0b8fb79f4b1c9a6c52bee85ae555, State: Initialized, Role: FOLLOWER
I20260810 06:24:46.523073 5673 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } } peers { permanent_uuid: "7f6e5871d8794034988627a01b1bc20f" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } } peers { permanent_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } }
I20260810 06:24:46.523476 5684 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } } peers { permanent_uuid: "7f6e5871d8794034988627a01b1bc20f" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } } peers { permanent_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } } }
I20260810 06:24:46.523607 5684 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:24:46.524057 5673 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 06:24:46.528276 5699 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:24:46.528349 5699 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:24:46.527093 5562 master_runner.cc:186] Error getting master registration for 127.31.204.254:36761: OK, instance_id { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" instance_seqno: 1786343086078879 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 06:24:46.530632 5562 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 06:24:46.532542 5553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } attempt: 3
Aug 10 06:24:46 dist-test-slave-q9h9 krb5kdc[5292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343086, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.253@KRBTEST.COM for kudu/127.31.204.252@KRBTEST.COM
I20260810 06:24:46.537751 5436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } attempt: 3
I20260810 06:24:46.539696 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.532891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:42169 (local address 127.0.0.1:40694)
0810 06:24:46.532983 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.532996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:46.533064 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:46.533269 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:46.533271 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:46.533413 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:46.533479 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:46.533484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.534171 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.534175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:46.534658 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:46.534665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.535070 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.535073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:46.535150 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.535509 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:46.535522 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:46.536532 (+ 1010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:46.538291 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:46.538294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:46.538297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:46.538523 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:46.538802 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:46.538805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:46.538807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:46.538889 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:46.539250 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:46.539253 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:46.539338 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 06:24:46.539433 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 06:24:46.539546 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 06:24:46.540132 5679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.532973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40694 (local address 127.31.204.252:42169)
0810 06:24:46.533065 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.533068 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.533081 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:24:46.533131 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.533134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.533170 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.533239 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.533576 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.534079 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.534756 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.534978 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.535059 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.536650 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:46.536654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:46.536656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:46.536680 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:46.537899 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.538624 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.538627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.538628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.538706 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.538970 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.538973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.538975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.539118 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:46.539204 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:46.539935 (+ 731us) server_negotiation.cc:299] Negotiation successful
0810 06:24:46.540038 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 06:24:46.542752 5553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f: Bootstrap starting.
Aug 10 06:24:46 dist-test-slave-q9h9 krb5kdc[5292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343086, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.204.254@KRBTEST.COM for kudu/127.31.204.252@KRBTEST.COM
I20260810 06:24:46.545197 5553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:46.545991 5553 log.cc:824] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:46.546486 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.538084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.204.252:42169 (local address 127.0.0.1:40710)
0810 06:24:46.538187 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:46.538200 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:46.538270 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:46.538785 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:46.538788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:46.538960 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:46.539048 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:46.539055 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.540041 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.540044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:46.540850 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:46.540861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.541376 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.541380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:46.541460 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.541859 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:46.541872 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:46.543185 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:46.545029 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:46.545032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:46.545034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:46.545269 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:46.545549 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:46.545552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:46.545554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:46.545632 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:46.546010 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:46.546036 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:46.546123 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 06:24:46.546218 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 06:24:46.546340 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 06:24:46.546866 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.538321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40710 (local address 127.31.204.252:42169)
0810 06:24:46.538571 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.538580 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.538591 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:24:46.538621 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.538623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.538663 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.538753 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.539160 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.539906 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.540970 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.541271 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.541362 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.543316 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:46.543320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:46.543323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:46.543347 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:46.544939 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.545355 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.545358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.545360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.545452 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.545717 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.545720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.545722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.545918 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:46.546011 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:46.546294 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 06:24:46.546405 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260810 06:24:46.547935 5553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f: No bootstrap required, opened a new log
I20260810 06:24:46.548976 5436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e: Bootstrap starting.
I20260810 06:24:46.550771 5436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:46.551363 5436 log.cc:824] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:46.551573 5553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } } peers { permanent_uuid: "7f6e5871d8794034988627a01b1bc20f" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } } peers { permanent_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } }
I20260810 06:24:46.551769 5553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6e5871d8794034988627a01b1bc20f, State: Initialized, Role: FOLLOWER
I20260810 06:24:46.552053 5553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } } peers { permanent_uuid: "7f6e5871d8794034988627a01b1bc20f" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } } peers { permanent_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } }
I20260810 06:24:46.552487 5707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } } peers { permanent_uuid: "7f6e5871d8794034988627a01b1bc20f" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } } peers { permanent_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } } }
I20260810 06:24:46.552667 5707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:24:46.553067 5436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e: No bootstrap required, opened a new log
I20260810 06:24:46.553227 5553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:46.555644 5436 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } } peers { permanent_uuid: "7f6e5871d8794034988627a01b1bc20f" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } } peers { permanent_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } }
I20260810 06:24:46.555862 5436 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9c8a3efdaa4463bb30e39dc8a05f2e, State: Initialized, Role: FOLLOWER
I20260810 06:24:46.556146 5436 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } } peers { permanent_uuid: "7f6e5871d8794034988627a01b1bc20f" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } } peers { permanent_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } }
I20260810 06:24:46.557313 5436 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [sys.catalog]: configured and running, proceeding with master startup.
W20260810 06:24:46.557996 5722 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:24:46.558174 5722 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:24:46.559513 5390 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 06:24:46.559623 5390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:46.560726 5390 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } } peers { permanent_uuid: "7f6e5871d8794034988627a01b1bc20f" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } } peers { permanent_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } }
I20260810 06:24:46.561545 5390 leader_election.cc:288] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [CANDIDATE]: Term 1 election: Requested vote from peers 7f6e5871d8794034988627a01b1bc20f (127.31.204.253:44099), a5ee0b8fb79f4b1c9a6c52bee85ae555 (127.31.204.252:42169)
I20260810 06:24:46.562290 5628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555"
I20260810 06:24:46.562419 5511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7f6e5871d8794034988627a01b1bc20f"
I20260810 06:24:46.562510 5628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:46.562598 5511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:46.556389 5447 master_runner.cc:186] Error getting master registration for 127.31.204.254:36761: OK, instance_id { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" instance_seqno: 1786343086078879 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 06:24:46.562975 5447 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 06:24:46.563656 5511 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa9c8a3efdaa4463bb30e39dc8a05f2e in term 1.
I20260810 06:24:46.564142 5628 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa9c8a3efdaa4463bb30e39dc8a05f2e in term 1.
I20260810 06:24:46.564020 5353 leader_election.cc:302] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f6e5871d8794034988627a01b1bc20f, fa9c8a3efdaa4463bb30e39dc8a05f2e; no voters:
I20260810 06:24:46.564960 5755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:46.565198 5755 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [term 1 LEADER]: Becoming Leader. State: Replica: fa9c8a3efdaa4463bb30e39dc8a05f2e, State: Running, Role: LEADER
I20260810 06:24:46.565541 5755 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } } peers { permanent_uuid: "7f6e5871d8794034988627a01b1bc20f" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } } peers { permanent_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } }
I20260810 06:24:46.569759 5724 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } } peers { permanent_uuid: "7f6e5871d8794034988627a01b1bc20f" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } } peers { permanent_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } } }
I20260810 06:24:46.569926 5724 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:46.570185 5724 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa9c8a3efdaa4463bb30e39dc8a05f2e. Latest consensus state: current_term: 1 leader_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } } peers { permanent_uuid: "7f6e5871d8794034988627a01b1bc20f" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } } peers { permanent_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } } }
I20260810 06:24:46.570343 5724 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [sys.catalog]: This master's current role is: LEADER
W20260810 06:24:46.562047 5745 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:24:46.573004 5745 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:24:46.574446 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.564908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35620 (local address 127.31.204.254:36761)
0810 06:24:46.565188 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.565192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.565203 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:24:46.565247 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.565250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.565288 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.565365 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.566289 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.567000 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.568160 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.568494 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.568566 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.570302 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:46.570306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:46.570309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:46.570333 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:46.571912 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.572959 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.572962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.572964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.573125 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.573438 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.573441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.573443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.573631 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:46.573749 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:46.574201 (+ 452us) server_negotiation.cc:299] Negotiation successful
0810 06:24:46.574311 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224}
W20260810 06:24:46.575596 5356 tablet.cc:2369] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:24:46.578318 5761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:46.579212 5761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:46.582844 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.565082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40718 (local address 127.31.204.252:42169)
0810 06:24:46.565174 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.565178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.574240 (+ 9062us) server_negotiation.cc:407] Connection header received
0810 06:24:46.574405 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.574408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.574449 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.574496 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.575442 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.576175 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.577304 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.577688 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.577779 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.579451 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:46.579455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:46.579458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:46.579482 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:46.581023 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.581512 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.581516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.581517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.581603 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.581915 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.581918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.581919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.582139 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:46.582246 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:46.582601 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 06:24:46.582709 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 06:24:46.586527 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:46.564538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.31.204.253:44099)
0810 06:24:46.564631 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.564635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.573322 (+ 8687us) server_negotiation.cc:407] Connection header received
0810 06:24:46.573496 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.573499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.573542 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.573589 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.574565 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.575274 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.576294 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:46.576603 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:46.576671 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:46.582985 (+ 6314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:46.582989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:46.582992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:46.583016 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:46.584566 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.585033 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.585036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.585038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.585124 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:46.585420 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:46.585423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:46.585425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:46.585609 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:46.585711 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:46.586280 (+ 569us) server_negotiation.cc:299] Negotiation successful
0810 06:24:46.586387 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 06:24:46.591149 5628 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555 [term 1 FOLLOWER]: Refusing update from remote peer fa9c8a3efdaa4463bb30e39dc8a05f2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:24:46.591374 5511 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f [term 1 FOLLOWER]: Refusing update from remote peer fa9c8a3efdaa4463bb30e39dc8a05f2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:24:46.591856 5758 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:24:46.592249 5758 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6e5871d8794034988627a01b1bc20f" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:24:46.595757 5707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa9c8a3efdaa4463bb30e39dc8a05f2e. Latest consensus state: current_term: 1 leader_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } } peers { permanent_uuid: "7f6e5871d8794034988627a01b1bc20f" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } } peers { permanent_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } } }
I20260810 06:24:46.595870 5707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:24:46.595989 5684 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa9c8a3efdaa4463bb30e39dc8a05f2e. Latest consensus state: current_term: 1 leader_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } } peers { permanent_uuid: "7f6e5871d8794034988627a01b1bc20f" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } } peers { permanent_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } } }
I20260810 06:24:46.596088 5684 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:24:46.605983 5762 mvcc.cc:205] Tried to move back new op lower bound from 7316861282657984512 to 7316861282576031744. Current Snapshot: MvccSnapshot[applied={T|T < 7316861282657984512}]
I20260810 06:24:46.607589 5758 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } } peers { permanent_uuid: "7f6e5871d8794034988627a01b1bc20f" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } } peers { permanent_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } } }
I20260810 06:24:46.607705 5758 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:46.608879 5684 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } } peers { permanent_uuid: "7f6e5871d8794034988627a01b1bc20f" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } } peers { permanent_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } } }
I20260810 06:24:46.608999 5684 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:24:46.609589 5758 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } } peers { permanent_uuid: "7f6e5871d8794034988627a01b1bc20f" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } } peers { permanent_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } } }
I20260810 06:24:46.609699 5758 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:46.610566 5707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9c8a3efdaa4463bb30e39dc8a05f2e" member_type: VOTER last_known_addr { host: "127.31.204.254" port: 36761 } } peers { permanent_uuid: "7f6e5871d8794034988627a01b1bc20f" member_type: VOTER last_known_addr { host: "127.31.204.253" port: 44099 } } peers { permanent_uuid: "a5ee0b8fb79f4b1c9a6c52bee85ae555" member_type: VOTER last_known_addr { host: "127.31.204.252" port: 42169 } } }
I20260810 06:24:46.610682 5707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:24:46.611523 5763 mvcc.cc:205] Tried to move back new op lower bound from 7316861282657984512 to 7316861282576031744. Current Snapshot: MvccSnapshot[applied={T|T < 7316861282657984512}]
I20260810 06:24:46.617051 5761 catalog_manager.cc:1347] Generated new cluster ID: 54d8f4af7b8643a7a8892d403d4d15f7
I20260810 06:24:46.617122 5761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:46.618223 5765 mvcc.cc:205] Tried to move back new op lower bound from 7316861282657984512 to 7316861282576031744. Current Snapshot: MvccSnapshot[applied={T|T < 7316861282657984512}]
I20260810 06:24:46.671741 5761 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:46.672971 5761 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:46.687676 5761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa9c8a3efdaa4463bb30e39dc8a05f2e: Generated new TSK 0
I20260810 06:24:46.688448 5761 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 06:24:46.717115 5443 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:46.714333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.31.204.254:36761)
0810 06:24:46.714460 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.714465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.716035 (+ 1570us) server_negotiation.cc:407] Connection header received
0810 06:24:46.716245 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.716249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.716303 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.716357 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.716989 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35630: BlockingRecv error: recv got EOF from 127.0.0.1:35630 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 06:24:46.718509 5559 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:46.715110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58232 (local address 127.31.204.253:44099)
0810 06:24:46.715225 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.715229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.717572 (+ 2343us) server_negotiation.cc:407] Connection header received
0810 06:24:46.717742 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.717745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.717807 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.717857 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.718391 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58232: BlockingRecv error: recv got EOF from 127.0.0.1:58232 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260810 06:24:46.719871 5701 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:46.716152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40728 (local address 127.31.204.252:42169)
0810 06:24:46.716274 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.716278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.718981 (+ 2703us) server_negotiation.cc:407] Connection header received
0810 06:24:46.719149 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.719152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.719204 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.719253 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.719743 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40728: BlockingRecv error: recv got EOF from 127.0.0.1:40728 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 06:24:46.724578 5559 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:46.722802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.31.204.253:44099)
0810 06:24:46.723344 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.723348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.723360 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:24:46.723428 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.723431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.723496 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.723563 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.724481 (+ 918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58238: BlockingRecv error: recv got EOF from 127.0.0.1:58238 (error 108)
Metrics: {"server-negotiator.queue_time_us":467}
W20260810 06:24:46.732297 5559 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:46.730766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58248 (local address 127.31.204.253:44099)
0810 06:24:46.730869 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.730873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.731415 (+ 542us) server_negotiation.cc:407] Connection header received
0810 06:24:46.731599 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.731601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.731653 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.731700 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.732190 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58248: BlockingRecv error: recv got EOF from 127.0.0.1:58248 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 06:24:46.737942 5559 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:46.736430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58252 (local address 127.31.204.253:44099)
0810 06:24:46.736532 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.736536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.737055 (+ 519us) server_negotiation.cc:407] Connection header received
0810 06:24:46.737238 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.737241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.737306 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.737359 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.737850 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58252: BlockingRecv error: recv got EOF from 127.0.0.1:58252 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 06:24:46.745595 5559 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:46.744020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.31.204.253:44099)
0810 06:24:46.744148 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.744152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.744734 (+ 582us) server_negotiation.cc:407] Connection header received
0810 06:24:46.744909 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.744912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.744957 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.745007 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.745503 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58266: BlockingRecv error: recv got EOF from 127.0.0.1:58266 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 06:24:46.756877 5559 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:46.755837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.31.204.253:44099)
0810 06:24:46.755932 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.755935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.756142 (+ 207us) server_negotiation.cc:407] Connection header received
0810 06:24:46.756276 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.756279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.756327 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.756406 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.756785 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58280: BlockingRecv error: recv got EOF from 127.0.0.1:58280 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260810 06:24:46.775914 5559 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:46.774841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58288 (local address 127.31.204.253:44099)
0810 06:24:46.774963 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.774967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.775139 (+ 172us) server_negotiation.cc:407] Connection header received
0810 06:24:46.775295 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.775298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.775343 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.775405 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.775822 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58288: BlockingRecv error: recv got EOF from 127.0.0.1:58288 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 06:24:46.810806 5559 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:46.809780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58298 (local address 127.31.204.253:44099)
0810 06:24:46.809923 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.809927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.810056 (+ 129us) server_negotiation.cc:407] Connection header received
0810 06:24:46.810203 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.810205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.810252 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.810326 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.810689 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58298: BlockingRecv error: recv got EOF from 127.0.0.1:58298 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260810 06:24:46.877864 5559 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:46.876736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.31.204.253:44099)
0810 06:24:46.876888 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:46.876893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:46.877003 (+ 110us) server_negotiation.cc:407] Connection header received
0810 06:24:46.877163 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:46.877166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:46.877215 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:46.877284 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:46.877743 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58300: BlockingRecv error: recv got EOF from 127.0.0.1:58300 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 06:24:47.009037 5559 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:47.007906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58302 (local address 127.31.204.253:44099)
0810 06:24:47.008068 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:47.008072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:47.008189 (+ 117us) server_negotiation.cc:407] Connection header received
0810 06:24:47.008344 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:47.008347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:47.008398 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:47.008464 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:47.008935 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58302: BlockingRecv error: recv got EOF from 127.0.0.1:58302 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 06:24:47.268253 5559 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:24:47.267126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58316 (local address 127.31.204.253:44099)
0810 06:24:47.267269 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:47.267274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:47.267342 (+ 68us) server_negotiation.cc:407] Connection header received
0810 06:24:47.267522 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:47.267524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:47.267575 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:47.267646 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:47.268128 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58316: BlockingRecv error: recv got EOF from 127.0.0.1:58316 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 06:24:47.529614 5699 catalog_manager.cc:1259] Loaded cluster ID: 54d8f4af7b8643a7a8892d403d4d15f7
I20260810 06:24:47.529707 5699 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555: loading cluster ID for follower catalog manager: success
I20260810 06:24:47.531406 5699 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555: acquiring CA information for follower catalog manager: success
I20260810 06:24:47.532536 5699 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a5ee0b8fb79f4b1c9a6c52bee85ae555: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 06:24:47.559675 5722 catalog_manager.cc:1259] Loaded cluster ID: 54d8f4af7b8643a7a8892d403d4d15f7
I20260810 06:24:47.559751 5722 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f: loading cluster ID for follower catalog manager: success
I20260810 06:24:47.561590 5722 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f: acquiring CA information for follower catalog manager: success
I20260810 06:24:47.562747 5722 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7f6e5871d8794034988627a01b1bc20f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 06:24:47.786006 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:47.782156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58332 (local address 127.31.204.253:44099)
0810 06:24:47.782398 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:47.782403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:47.782479 (+ 76us) server_negotiation.cc:407] Connection header received
0810 06:24:47.782624 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:47.782627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:47.782752 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:47.782857 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:24:47.783133 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:47.783648 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:47.784524 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:47.784753 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:47.784849 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:47.785177 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:24:47.785605 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:24:47.785713 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:47.785797 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 06:24:47.785856 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":86,"threads_started":1}
I20260810 06:24:47.791878 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:47.788743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40740 (local address 127.31.204.252:42169)
0810 06:24:47.788954 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:47.788958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:47.788969 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:24:47.789041 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:47.789043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:47.789147 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:47.789231 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:24:47.789396 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:47.789885 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:47.790717 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:47.790941 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:47.791017 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:47.791215 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:24:47.791569 (+ 354us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:24:47.791649 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:47.791678 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 06:24:47.791722 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":74,"threads_started":1}
I20260810 06:24:47.798388 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:47.794597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58336 (local address 127.31.204.253:44099)
0810 06:24:47.794700 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:47.794704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:47.795133 (+ 429us) server_negotiation.cc:407] Connection header received
0810 06:24:47.795253 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:47.795255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:47.795312 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:47.795375 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:24:47.795698 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:47.796416 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:47.797439 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:47.797652 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:47.797735 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:47.797975 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:24:47.798078 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:24:47.798141 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:47.798214 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 06:24:47.798261 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:24:47.798909 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:47.794338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35644 (local address 127.31.204.254:36761)
0810 06:24:47.794579 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:47.794583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:47.794597 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:24:47.794678 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:47.794681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:47.794787 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:47.794897 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:24:47.795481 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:47.796224 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:47.797333 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:47.797683 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:47.797785 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:47.798170 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:24:47.798478 (+ 308us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:24:47.798560 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:47.798695 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 06:24:47.798759 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
I20260810 06:24:47.799321 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:47.794976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40756 (local address 127.31.204.252:42169)
0810 06:24:47.795077 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:47.795081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:47.795161 (+ 80us) server_negotiation.cc:407] Connection header received
0810 06:24:47.795292 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:47.795295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:47.795338 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:47.795400 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:24:47.796516 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:47.796978 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:47.798035 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:47.798354 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:47.798451 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:47.798873 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:24:47.798943 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:24:47.799012 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:47.799153 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 06:24:47.799204 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 06:24:47.801729 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 5319
I20260810 06:24:47.815649 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 5447
I20260810 06:24:47.825959 32563 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DS5L1/build/debug/bin/kudu with pid 5562
2026-08-10T06:24:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3194 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6363 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (284126 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST